From sashak at voltaire.com Sun Mar 1 00:08:04 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 1 Mar 2009 10:08:04 +0200 Subject: [ofa-general] Re: [PATCH 2/10] infiniband-diags: Convert ibaddr to "new" ibmad interface In-Reply-To: <20090219190528.11c080f8.weiny2@llnl.gov> References: <20090219190528.11c080f8.weiny2@llnl.gov> Message-ID: <20090301080804.GH3936@sashak.voltaire.com> On 19:05 Thu 19 Feb , Ira Weiny wrote: > From 1ead0cdb05b159dbd3a89d2030870fc7326ec84d Mon Sep 17 00:00:00 2001 > From: Ira Weiny > Date: Thu, 19 Feb 2009 14:47:05 -0800 > Subject: [PATCH] infiniband-diags: Convert ibaddr to "new" ibmad interface > > > Signed-off-by: Ira Weiny > --- > infiniband-diags/src/ibaddr.c | 17 ++++++++++++----- > infiniband-diags/src/ibdiag_common.c | 3 ++- > 2 files changed, 14 insertions(+), 6 deletions(-) > > diff --git a/infiniband-diags/src/ibaddr.c b/infiniband-diags/src/ibaddr.c > index 9098699..bb22be9 100644 > --- a/infiniband-diags/src/ibaddr.c > +++ b/infiniband-diags/src/ibaddr.c > @@ -45,6 +45,8 @@ > > #include "ibdiag_common.h" > > +struct ibmad_port *srcport; > + > static int > ib_resolve_addr(ib_portid_t *portid, int portnum, int show_lid, int show_gid) > { > @@ -55,10 +57,10 @@ ib_resolve_addr(ib_portid_t *portid, int portnum, int show_lid, int show_gid) > ibmad_gid_t gid; > int lmc; > > - if (!smp_query(nodeinfo, portid, IB_ATTR_NODE_INFO, 0, 0)) > + if (!smp_query_via(nodeinfo, portid, IB_ATTR_NODE_INFO, 0, 0, srcport)) > return -1; > > - if (!smp_query(portinfo, portid, IB_ATTR_PORT_INFO, portnum, 0)) > + if (!smp_query_via(portinfo, portid, IB_ATTR_PORT_INFO, portnum, 0, srcport)) > return -1; > > mad_decode_field(portinfo, IB_PORT_LID_F, &portid->lid); > @@ -137,17 +139,22 @@ int main(int argc, char **argv) > if (!show_lid && !show_gid) > show_lid = show_gid = 1; > > - madrpc_init(ibd_ca, ibd_ca_port, mgmt_classes, 3); > + srcport = mad_rpc_open_port(ibd_ca, ibd_ca_port, mgmt_classes, 3); > + if (!srcport) > + IBERROR("Failed to open '%s' port '%d'", ibd_ca, ibd_ca_port); Here and in almost all other infiniband-diags patches: madrpc_init() in case of failures calls IBPANIC() which prints error message and calls exit(). Unlike this mad_rpc_open_port() sets errno and returns NULL, IBERROR() just prints error message - the program will continue execution... > > if (argc) { > - if (ib_resolve_portid_str(&portid, argv[0], ibd_dest_type, ibd_sm_id) < 0) > + if (ib_resolve_portid_str_via(&portid, argv[0], ibd_dest_type, > + ibd_sm_id, srcport) < 0) > IBERROR("can't resolve destination port %s", argv[0]); > } else { > - if (ib_resolve_self(&portid, &port, 0) < 0) > + if (ib_resolve_self_via(&portid, &port, 0, srcport) < 0) > IBERROR("can't resolve self port %s", argv[0]); > } > > if (ib_resolve_addr(&portid, port, show_lid, show_gid) < 0) > IBERROR("can't resolve requested address"); > + > + mad_rpc_close_port(srcport); > exit(0); > } > diff --git a/infiniband-diags/src/ibdiag_common.c b/infiniband-diags/src/ibdiag_common.c > index 5f2472d..609df69 100644 > --- a/infiniband-diags/src/ibdiag_common.c > +++ b/infiniband-diags/src/ibdiag_common.c > @@ -179,7 +179,8 @@ static int process_opt(int ch, char *optarg) > ibd_timeout = val; > break; > case 's': > - if (ib_resolve_portid_str(&sm_portid, optarg, IB_DEST_LID, 0) < 0) > + if (ib_resolve_portid_str_via(&sm_portid, optarg, IB_DEST_LID, > + 0, NULL) < 0) This may be not correct (not introduced by those patches) - then host is connected to multiple subnets we need to resolve SM LID on requested one, eg. using srcport... Sasha > IBERROR("cannot resolve SM destination port %s", optarg); > ibd_sm_id = &sm_portid; > break; > -- > 1.5.4.5 > From sashak at voltaire.com Sun Mar 1 00:10:10 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 1 Mar 2009 10:10:10 +0200 Subject: [ofa-general] Re: [PATCH 1/3 v2] opensm: Added io_guid_file and max_reverse_hops options In-Reply-To: <49AA3664.8090104@dev.mellanox.co.il> References: <49953C48.3030203@ext.bull.net> <20090228191921.GA3936@sashak.voltaire.com> <49A9A1E0.4050005@morey-chaisemartin.com> <20090228215645.GD3936@sashak.voltaire.com> <49AA3664.8090104@dev.mellanox.co.il> Message-ID: <20090301081010.GI3936@sashak.voltaire.com> On 09:16 Sun 01 Mar , Yevgeny Kliteynik wrote: > Sasha Khapyorsky wrote: >> On 21:43 Sat 28 Feb , Nicolas Morey-Chaisemartin wrote: >>> I tried to put my patches inline as Yevgeni advised me but it seems >>> thunderbird messes things up though I directly output git format-patch >>> into a thunderbird draft file. >>> I guess I'll stick to attachment from now on... >> Attached patches are not friendly for reviewing. Look at Thunderbird >> related section of: >> http://git.kernel.org/?p=git/git.git;a=blob_plain;f=Documentation/SubmittingPatches > > The Thunderbird section describes two options. > There's also a third option - the QuickText Thunderbird extension: > https://addons.mozilla.org/en-US/thunderbird/addon/640 > With this extension you will get the new bar when composing mail. > Go to "Other"->"Insert file as text" and insert the patch. Maybe this works (I'm not using Thunderbird), but patches were broken... Sasha From sashak at voltaire.com Sun Mar 1 00:16:17 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 1 Mar 2009 10:16:17 +0200 Subject: [ofa-general] [PATCH 1/2] libibmad: add PortXmtDataSL / PortRcvDataSL support In-Reply-To: <49AA3E52.30804@Voltaire.com> References: <3B25B2D61996446F88703F647919FC4E@amr.corp.intel.com> <49AA3E52.30804@Voltaire.com> Message-ID: <20090301081617.GJ3936@sashak.voltaire.com> On 09:50 Sun 01 Mar , Or Gerlitz wrote: > Sean Hefty wrote: > > Rather than continue to add more and more interfaces to the library, can we just > > export a couple of more generic calls? > > Hi Sasha, > > So how you'd like to get this done? should I just expose pma_query, pma_query_via, > performance_reset, etc through mad.h? It could be better (I didn't see the patch although). Also it would be nice (when possible) to add new field entries at end of ib_mad_f array - in this way libibmad still be backward compatible. Sasha From vlad at lists.openfabrics.org Sun Mar 1 03:19:42 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sun, 1 Mar 2009 03:19:42 -0800 (PST) Subject: [ofa-general] ofa_1_4_kernel 20090301-0200 daily build status Message-ID: <20090301111942.E4AD9E60ED0@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From tziporet at dev.mellanox.co.il Sun Mar 1 03:51:54 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Sun, 01 Mar 2009 13:51:54 +0200 Subject: ***SPAM*** Re: ***SPAM*** Re: [ofa-general] ib_reg_phys_mr( ) results in crash In-Reply-To: <7d5928b30902271101h589ad61cha59f626572a24802@mail.gmail.com> References: <7d5928b30902170650o234f586ax6e27bb82c46427b3@mail.gmail.com> <7d5928b30902191047o25c34462w4cc51d7b88b888c6@mail.gmail.com> <499E6826.704@sun.com> <7d5928b30902271101h589ad61cha59f626572a24802@mail.gmail.com> Message-ID: <49AA76DA.4010305@mellanox.co.il> neutron wrote: > It might be related to new ConnectX card (with mlx4_ib module). > > Now I tried the same program on a machine with only "mthca" card, it > succeeds without any problems. > > > I remember one guy in this list also reported a similar issue: > ib_phys_reg_mr( ) fails with mlx4 module. > > We have not implemented ib_phys_reg_mr in mlx4 You should use another method for registering memory in mlx4 Roland - maybe we should implement it Tziporet From ogerlitz at voltaire.com Sun Mar 1 04:37:54 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 1 Mar 2009 14:37:54 +0200 (IST) Subject: [ofa-general] [PATCH v2 0/4] generic perf query/reset api plus PortXmtDataSL/PortRcvDataSL impl Message-ID: Following the comments from the initial version, this patch series first exports a generic libibmad api to issue query and reset on performance counters and then implements PortXmtDataSL and PortRcvDataSL. Or. From ogerlitz at voltaire.com Sun Mar 1 04:39:02 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 1 Mar 2009 14:39:02 +0200 (IST) Subject: [ofa-general] [PATCH 1/4] generic libibmad perf query/reset api In-Reply-To: References: Message-ID: Export generic api to issue query and reset on performance counters, this will allow for easier integration of more counters (e.g QoS, CCA related). Signed-off-by: Or Gerlitz Index: management/libibmad/include/infiniband/mad.h =================================================================== --- management.orig/libibmad/include/infiniband/mad.h +++ management/libibmad/include/infiniband/mad.h @@ -781,43 +781,13 @@ int ib_resolve_self_via(ib_portid_t * po const void *srcport); /* gs.c */ -MAD_EXPORT uint8_t *perf_classportinfo_query(void *rcvbuf, ib_portid_t * dest, - int port, unsigned timeout); -MAD_EXPORT uint8_t *port_performance_query(void *rcvbuf, ib_portid_t * dest, - int port, unsigned timeout); -MAD_EXPORT uint8_t *port_performance_reset(void *rcvbuf, ib_portid_t * dest, - int port, unsigned mask, - unsigned timeout); -MAD_EXPORT uint8_t *port_performance_ext_query(void *rcvbuf, ib_portid_t * dest, - int port, unsigned timeout); -MAD_EXPORT uint8_t *port_performance_ext_reset(void *rcvbuf, ib_portid_t * dest, - int port, unsigned mask, - unsigned timeout); -MAD_EXPORT uint8_t *port_samples_control_query(void *rcvbuf, ib_portid_t * dest, - int port, unsigned timeout); -MAD_EXPORT uint8_t *port_samples_result_query(void *rcvbuf, ib_portid_t * dest, - int port, unsigned timeout); - -uint8_t *perf_classportinfo_query_via(void *rcvbuf, ib_portid_t * dest, - int port, unsigned timeout, - const void *srcport); -uint8_t *port_performance_query_via(void *rcvbuf, ib_portid_t * dest, int port, - unsigned timeout, const void *srcport); -uint8_t *port_performance_reset_via(void *rcvbuf, ib_portid_t * dest, int port, - unsigned mask, unsigned timeout, - const void *srcport); -uint8_t *port_performance_ext_query_via(void *rcvbuf, ib_portid_t * dest, - int port, unsigned timeout, - const void *srcport); -uint8_t *port_performance_ext_reset_via(void *rcvbuf, ib_portid_t * dest, - int port, unsigned mask, - unsigned timeout, const void *srcport); -uint8_t *port_samples_control_query_via(void *rcvbuf, ib_portid_t * dest, - int port, unsigned timeout, - const void *srcport); -uint8_t *port_samples_result_query_via(void *rcvbuf, ib_portid_t * dest, - int port, unsigned timeout, - const void *srcport); +MAD_EXPORT uint8_t *pma_query_via(void *rcvbuf, ib_portid_t * dest, int port, + unsigned timeout, unsigned id, + const void *srcport); + +MAD_EXPORT uint8_t *performance_reset_via(void *rcvbuf, ib_portid_t * dest, + int port, unsigned mask, unsigned timeout, + unsigned id, const void *srcport); /* dump.c */ MAD_EXPORT ib_mad_dump_fn mad_dump_int, mad_dump_uint, mad_dump_hex, mad_dump_rhex, Index: management/libibmad/src/gs.c =================================================================== --- management.orig/libibmad/src/gs.c +++ management/libibmad/src/gs.c @@ -45,9 +45,9 @@ #undef DEBUG #define DEBUG if (ibdebug) IBWARN -static uint8_t *pma_query_via(void *rcvbuf, ib_portid_t * dest, int port, - unsigned timeout, unsigned id, - const void *srcport) +uint8_t *pma_query_via(void *rcvbuf, ib_portid_t * dest, int port, + unsigned timeout, unsigned id, + const void *srcport) { ib_rpc_t rpc = { 0 }; int lid = dest->lid; @@ -81,42 +81,9 @@ static uint8_t *pma_query_via(void *rcvb } } -uint8_t *pma_query(void *rcvbuf, ib_portid_t * dest, int port, unsigned timeout, - unsigned id) -{ - return pma_query_via(rcvbuf, dest, port, timeout, id, NULL); -} - -uint8_t *perf_classportinfo_query_via(void *rcvbuf, ib_portid_t * dest, - int port, unsigned timeout, - const void *srcport) -{ - return pma_query_via(rcvbuf, dest, port, timeout, CLASS_PORT_INFO, - srcport); -} - -uint8_t *perf_classportinfo_query(void *rcvbuf, ib_portid_t * dest, int port, - unsigned timeout) -{ - return pma_query(rcvbuf, dest, port, timeout, CLASS_PORT_INFO); -} - -uint8_t *port_performance_query_via(void *rcvbuf, ib_portid_t * dest, int port, - unsigned timeout, const void *srcport) -{ - return pma_query_via(rcvbuf, dest, port, timeout, - IB_GSI_PORT_COUNTERS, srcport); -} - -uint8_t *port_performance_query(void *rcvbuf, ib_portid_t * dest, int port, - unsigned timeout) -{ - return pma_query(rcvbuf, dest, port, timeout, IB_GSI_PORT_COUNTERS); -} - -static uint8_t *performance_reset_via(void *rcvbuf, ib_portid_t * dest, - int port, unsigned mask, unsigned timeout, - unsigned id, const void *srcport) +uint8_t *performance_reset_via(void *rcvbuf, ib_portid_t * dest, + int port, unsigned mask, unsigned timeout, + unsigned id, const void *srcport) { ib_rpc_t rpc = { 0 }; int lid = dest->lid; @@ -156,84 +123,3 @@ static uint8_t *performance_reset_via(vo return madrpc(&rpc, dest, rcvbuf, rcvbuf); } } - -static uint8_t *performance_reset(void *rcvbuf, ib_portid_t * dest, int port, - unsigned mask, unsigned timeout, unsigned id) -{ - return performance_reset_via(rcvbuf, dest, port, mask, timeout, - id, NULL); -} - -uint8_t *port_performance_reset_via(void *rcvbuf, ib_portid_t * dest, int port, - unsigned mask, unsigned timeout, - const void *srcport) -{ - return performance_reset_via(rcvbuf, dest, port, mask, timeout, - IB_GSI_PORT_COUNTERS, srcport); -} - -uint8_t *port_performance_reset(void *rcvbuf, ib_portid_t * dest, int port, - unsigned mask, unsigned timeout) -{ - return performance_reset(rcvbuf, dest, port, mask, timeout, - IB_GSI_PORT_COUNTERS); -} - -uint8_t *port_performance_ext_query_via(void *rcvbuf, ib_portid_t * dest, - int port, unsigned timeout, - const void *srcport) -{ - return pma_query_via(rcvbuf, dest, port, timeout, - IB_GSI_PORT_COUNTERS_EXT, srcport); -} - -uint8_t *port_performance_ext_query(void *rcvbuf, ib_portid_t * dest, int port, - unsigned timeout) -{ - return pma_query(rcvbuf, dest, port, timeout, IB_GSI_PORT_COUNTERS_EXT); -} - -uint8_t *port_performance_ext_reset_via(void *rcvbuf, ib_portid_t * dest, - int port, unsigned mask, - unsigned timeout, const void *srcport) -{ - return performance_reset_via(rcvbuf, dest, port, mask, timeout, - IB_GSI_PORT_COUNTERS_EXT, srcport); -} - -uint8_t *port_performance_ext_reset(void *rcvbuf, ib_portid_t * dest, int port, - unsigned mask, unsigned timeout) -{ - return performance_reset(rcvbuf, dest, port, mask, timeout, - IB_GSI_PORT_COUNTERS_EXT); -} - -uint8_t *port_samples_control_query_via(void *rcvbuf, ib_portid_t * dest, - int port, unsigned timeout, - const void *srcport) -{ - return pma_query_via(rcvbuf, dest, port, timeout, - IB_GSI_PORT_SAMPLES_CONTROL, srcport); -} - -uint8_t *port_samples_control_query(void *rcvbuf, ib_portid_t * dest, int port, - unsigned timeout) -{ - return pma_query(rcvbuf, dest, port, timeout, - IB_GSI_PORT_SAMPLES_CONTROL); -} - -uint8_t *port_samples_result_query_via(void *rcvbuf, ib_portid_t * dest, - int port, unsigned timeout, - const void *srcport) -{ - return pma_query_via(rcvbuf, dest, port, timeout, - IB_GSI_PORT_SAMPLES_RESULT, srcport); -} - -uint8_t *port_samples_result_query(void *rcvbuf, ib_portid_t * dest, int port, - unsigned timeout) -{ - return pma_query(rcvbuf, dest, port, timeout, - IB_GSI_PORT_SAMPLES_RESULT); -} Index: management/libibmad/src/libibmad.map =================================================================== --- management.orig/libibmad/src/libibmad.map +++ management/libibmad/src/libibmad.map @@ -40,13 +40,8 @@ IBMAD_1.3 { mad_set_field64; mad_get_array; mad_set_array; - perf_classportinfo_query; - port_performance_query; - port_performance_reset; - port_performance_ext_query; - port_performance_ext_reset; - port_samples_control_query; - port_samples_result_query; + pma_query_via; + performance_reset_via; mad_build_pkt; mad_decode_field; mad_encode; @@ -97,11 +92,5 @@ IBMAD_1.3 { ib_resolve_portid_str_via; ib_resolve_self_via; perf_classportinfo_query_via; - port_performance_query_via; - port_performance_reset_via; - port_performance_ext_query_via; - port_performance_ext_reset_via; - port_samples_control_query_via; - port_samples_result_query_via; local: *; }; From ogerlitz at voltaire.com Sun Mar 1 04:39:48 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 1 Mar 2009 14:39:48 +0200 (IST) Subject: [ofa-general] [PATCH 1/4] port perfquery to the generic libibmad performance api In-Reply-To: References: Message-ID: port perfquery to the generic libibmad api for performance counters query and reset. Signed-off-by: Or Gerlitz Index: management/infiniband-diags/src/perfquery.c =================================================================== --- management.orig/infiniband-diags/src/perfquery.c +++ management/infiniband-diags/src/perfquery.c @@ -269,7 +269,8 @@ static void dump_perfcounters(int extend char buf[1024]; if (extended != 1) { - if (!port_performance_query(pc, portid, port, timeout)) + if (!pma_query_via(pc, portid, port, timeout, + IB_GSI_PORT_COUNTERS, NULL)) IBERROR("perfquery"); if (!(cap_mask & 0x1000)) { /* if PortCounters:PortXmitWait not suppported clear this counter */ @@ -284,7 +285,8 @@ static void dump_perfcounters(int extend if (!(cap_mask & 0x200)) /* 1.2 errata: bit 9 is extended counter support */ IBWARN("PerfMgt ClassPortInfo 0x%x extended counters not indicated\n", cap_mask); - if (!port_performance_ext_query(pc, portid, port, timeout)) + if (!pma_query_via(pc, portid, port, timeout, + IB_GSI_PORT_COUNTERS_EXT, NULL)) IBERROR("perfextquery"); if (aggregate) aggregate_perfcounters_ext(); @@ -299,10 +301,12 @@ static void dump_perfcounters(int extend static void reset_counters(int extended, int timeout, int mask, ib_portid_t *portid, int port) { if (extended != 1) { - if (!port_performance_reset(pc, portid, port, mask, timeout)) + if (!performance_reset_via(pc, portid, port, mask, timeout, + IB_GSI_PORT_COUNTERS, NULL)) IBERROR("perf reset"); } else { - if (!port_performance_ext_reset(pc, portid, port, mask, timeout)) + if (!performance_reset_via(pc, portid, port, mask, timeout, + IB_GSI_PORT_COUNTERS_EXT, NULL)) IBERROR("perf ext reset"); } } @@ -393,7 +397,8 @@ int main(int argc, char **argv) } /* PerfMgt ClassPortInfo is a required attribute */ - if (!perf_classportinfo_query(pc, &portid, port, ibd_timeout)) + if (!pma_query_via(pc, &portid, port, ibd_timeout, + CLASS_PORT_INFO, NULL)) IBERROR("classportinfo query"); /* ClassPortInfo should be supported as part of libibmad */ memcpy(&cap_mask, pc + 2, sizeof(cap_mask)); /* CapabilityMask */ From ogerlitz at voltaire.com Sun Mar 1 04:40:32 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 1 Mar 2009 14:40:32 +0200 (IST) Subject: [ofa-general] [PATCH v2 3/4] libimad implementation of PortXmtDataSL and PortRcvDataSL In-Reply-To: References: Message-ID: libimad implementation of PortXmtDataSL (IBA A13.6.5) / PortRcvDataSL (IBA A13.6.6) reading and resetting Signed-off-by: Or Gerlitz Index: management/libibmad/include/infiniband/mad.h =================================================================== --- management.orig/libibmad/include/infiniband/mad.h +++ management/libibmad/include/infiniband/mad.h @@ -153,7 +153,8 @@ enum GSI_ATTR_ID { IB_GSI_PORT_SAMPLES_RESULT = 0x11, IB_GSI_PORT_COUNTERS = 0x12, IB_GSI_PORT_COUNTERS_EXT = 0x1D, - + IB_GSI_PORT_XMIT_DATA_SL = 0x36, + IB_GSI_PORT_RCV_DATA_SL = 0x37, IB_GSI_ATTR_LAST }; @@ -562,6 +563,44 @@ enum MAD_FIELDS { IB_CPI_TRAP_QP_F, IB_CPI_TRAP_QKEY_F, + IB_PC_XMT_DATA_SL_FIRST_F, + IB_PC_XMT_DATA_SL0_F = IB_PC_XMT_DATA_SL_FIRST_F, + IB_PC_XMT_DATA_SL1_F, + IB_PC_XMT_DATA_SL2_F, + IB_PC_XMT_DATA_SL3_F, + IB_PC_XMT_DATA_SL4_F, + IB_PC_XMT_DATA_SL5_F, + IB_PC_XMT_DATA_SL6_F, + IB_PC_XMT_DATA_SL7_F, + IB_PC_XMT_DATA_SL8_F, + IB_PC_XMT_DATA_SL9_F, + IB_PC_XMT_DATA_SL10_F, + IB_PC_XMT_DATA_SL11_F, + IB_PC_XMT_DATA_SL12_F, + IB_PC_XMT_DATA_SL13_F, + IB_PC_XMT_DATA_SL14_F, + IB_PC_XMT_DATA_SL15_F, + IB_PC_XMT_DATA_SL_LAST_F, + + IB_PC_RCV_DATA_SL_FIRST_F, + IB_PC_RCV_DATA_SL0_F = IB_PC_RCV_DATA_SL_FIRST_F, + IB_PC_RCV_DATA_SL1_F, + IB_PC_RCV_DATA_SL2_F, + IB_PC_RCV_DATA_SL3_F, + IB_PC_RCV_DATA_SL4_F, + IB_PC_RCV_DATA_SL5_F, + IB_PC_RCV_DATA_SL6_F, + IB_PC_RCV_DATA_SL7_F, + IB_PC_RCV_DATA_SL8_F, + IB_PC_RCV_DATA_SL9_F, + IB_PC_RCV_DATA_SL10_F, + IB_PC_RCV_DATA_SL11_F, + IB_PC_RCV_DATA_SL12_F, + IB_PC_RCV_DATA_SL13_F, + IB_PC_RCV_DATA_SL14_F, + IB_PC_RCV_DATA_SL15_F, + IB_PC_RCV_DATA_SL_LAST_F, + IB_FIELD_LAST_ /* must be last */ }; @@ -800,7 +839,8 @@ MAD_EXPORT ib_mad_dump_fn mad_dump_mtu, mad_dump_vlcap, mad_dump_opervls, mad_dump_node_type, mad_dump_sltovl, mad_dump_vlarbitration, mad_dump_nodedesc, mad_dump_nodeinfo, mad_dump_portinfo, - mad_dump_switchinfo, mad_dump_perfcounters, mad_dump_perfcounters_ext; + mad_dump_switchinfo, mad_dump_perfcounters, mad_dump_perfcounters_ext, + mad_dump_perfcounters_xmt_sl, mad_dump_perfcounters_rcv_sl; extern int ibdebug; Index: management/libibmad/src/fields.c =================================================================== --- management.orig/libibmad/src/fields.c +++ management/libibmad/src/fields.c @@ -402,6 +402,42 @@ static const ib_field_t ib_mad_f[] = { {BITSOFFS(520, 24), "TrapQP", mad_dump_hex}, {544, 32, "TrapQKey", mad_dump_hex}, + {32, 32, "XmtDataSL0", mad_dump_uint}, + {64, 32, "XmtDataSL1", mad_dump_uint}, + {96, 32, "XmtDataSL2", mad_dump_uint}, + {128, 32, "XmtDataSL3", mad_dump_uint}, + {160, 32, "XmtDataSL4", mad_dump_uint}, + {196, 32, "XmtDataSL5", mad_dump_uint}, + {224, 32, "XmtDataSL6", mad_dump_uint}, + {256, 32, "XmtDataSL7", mad_dump_uint}, + {288, 32, "XmtDataSL8", mad_dump_uint}, + {320, 32, "XmtDataSL9", mad_dump_uint}, + {352, 32, "XmtDataSL10", mad_dump_uint}, + {384, 32, "XmtDataSL11", mad_dump_uint}, + {416, 32, "XmtDataSL12", mad_dump_uint}, + {448, 32, "XmtDataSL13", mad_dump_uint}, + {480, 32, "XmtDataSL14", mad_dump_uint}, + {512, 32, "XmtDataSL15", mad_dump_uint}, + {0, 0}, /* IB_PC_XMT_DATA_SL_LAST_F */ + + {32, 32, "RcvDataSL0", mad_dump_uint}, + {64, 32, "RcvDataSL1", mad_dump_uint}, + {96, 32, "RcvDataSL2", mad_dump_uint}, + {128, 32, "RcvDataSL3", mad_dump_uint}, + {160, 32, "RcvDataSL4", mad_dump_uint}, + {196, 32, "RcvDataSL5", mad_dump_uint}, + {224, 32, "RcvDataSL6", mad_dump_uint}, + {256, 32, "RcvDataSL7", mad_dump_uint}, + {288, 32, "RcvDataSL8", mad_dump_uint}, + {320, 32, "RcvDataSL9", mad_dump_uint}, + {352, 32, "RcvDataSL10", mad_dump_uint}, + {384, 32, "RcvDataSL11", mad_dump_uint}, + {416, 32, "RcvDataSL12", mad_dump_uint}, + {448, 32, "RcvDataSL13", mad_dump_uint}, + {480, 32, "RcvDataSL14", mad_dump_uint}, + {512, 32, "RcvDataSL15", mad_dump_uint}, + {0, 0}, /* IB_PC_RCV_DATA_SL_LAST_F */ + {0, 0} /* IB_FIELD_LAST_ */ }; Index: management/libibmad/src/dump.c =================================================================== --- management.orig/libibmad/src/dump.c +++ management/libibmad/src/dump.c @@ -699,6 +699,16 @@ void mad_dump_perfcounters_ext(char *buf _dump_fields(buf, bufsz, val, IB_PC_EXT_FIRST_F, IB_PC_EXT_LAST_F); } +void mad_dump_perfcounters_xmt_sl(char *buf, int bufsz, void *val, int valsz) +{ + _dump_fields(buf, bufsz, val, IB_PC_XMT_DATA_SL_FIRST_F, IB_PC_XMT_DATA_SL_LAST_F); +} + +void mad_dump_perfcounters_rcv_sl(char *buf, int bufsz, void *val, int valsz) +{ + _dump_fields(buf, bufsz, val, IB_PC_RCV_DATA_SL_FIRST_F, IB_PC_RCV_DATA_SL_LAST_F); +} + void xdump(FILE * file, char *msg, void *p, int size) { #define HEX(x) ((x) < 10 ? '0' + (x) : 'a' + ((x) -10)) Index: management/libibmad/src/libibmad.map =================================================================== --- management.orig/libibmad/src/libibmad.map +++ management/libibmad/src/libibmad.map @@ -22,6 +22,8 @@ IBMAD_1.3 { mad_dump_opervls; mad_dump_perfcounters; mad_dump_perfcounters_ext; + mad_dump_perfcounters_xmt_sl; + mad_dump_perfcounters_rcv_sl; mad_dump_physportstate; mad_dump_portcapmask; mad_dump_portinfo; From ogerlitz at voltaire.com Sun Mar 1 04:41:51 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 1 Mar 2009 14:41:51 +0200 (IST) Subject: [ofa-general] [PATCH v2 4/4] perfquery PortXmtDataSL/PortRcvDataSL support In-Reply-To: References: Message-ID: perfquery PortXmtDataSL/PortRcvDataSL (IBA A13.6.5/6) support Signed-off-by: Or Gerlitz -- changes from V1: set command line args such for unused shortcuts Index: management/infiniband-diags/src/perfquery.c =================================================================== --- management.orig/infiniband-diags/src/perfquery.c +++ management/infiniband-diags/src/perfquery.c @@ -311,7 +311,55 @@ static void reset_counters(int extended, } } -static int reset, reset_only, all_ports, loop_ports, port, extended; +static int reset, reset_only, all_ports, loop_ports, port, extended, xmt_sl, rcv_sl; + +void xmt_sl_query(ib_portid_t *portid, int port, int mask) +{ + char buf[1024]; + + if (reset_only) { + if (!performance_reset_via(pc, portid, port, mask, ibd_timeout, + IB_GSI_PORT_XMIT_DATA_SL, NULL)) + IBERROR("perfslreset"); + return; + } + + if (!pma_query_via(pc, portid, port, ibd_timeout, + IB_GSI_PORT_XMIT_DATA_SL, NULL)) + IBERROR("perfslquery"); + + mad_dump_perfcounters_xmt_sl(buf, sizeof buf, pc, sizeof pc); + printf("# Port counters: %s port %d\n%s", portid2str(portid), port, buf); + + if(reset) + if (!performance_reset_via(pc, portid, port, mask, ibd_timeout, + IB_GSI_PORT_XMIT_DATA_SL, NULL)) + IBERROR("perfslreset"); +} + +void rcv_sl_query(ib_portid_t *portid, int port, int mask) +{ + char buf[1024]; + + if (reset_only) { + if (!performance_reset_via(pc, portid, port, mask, ibd_timeout, + IB_GSI_PORT_RCV_DATA_SL, NULL)) + IBERROR("perfslreset"); + return; + } + + if (!pma_query_via(pc, portid, port, ibd_timeout, + IB_GSI_PORT_RCV_DATA_SL, NULL)) + IBERROR("perfslquery"); + + mad_dump_perfcounters_rcv_sl(buf, sizeof buf, pc, sizeof pc); + printf("# Port counters: %s port %d\n%s", portid2str(portid), port, buf); + + if(reset) + if (!performance_reset_via(pc, portid, port, mask, ibd_timeout, + IB_GSI_PORT_RCV_DATA_SL, NULL)) + IBERROR("perfslreset"); +} static int process_opt(void *context, int ch, char *optarg) { @@ -319,6 +367,12 @@ static int process_opt(void *context, in case 'x': extended = 1; break; + case 'X': + xmt_sl = 1; + break; + case 'S': + rcv_sl = 1; + break; case 'a': all_ports++; port = ALL_PORTS; @@ -353,6 +407,8 @@ int main(int argc, char **argv) const struct ibdiag_opt opts[] = { { "extended", 'x', 0, NULL, "show extended port counters" }, + { "xmtsl", 'X', 0, NULL, "show Xmt SL port counters" }, + { "rcvsl", 'S', 0, NULL, "show Rcv SL port counters" }, { "all_ports", 'a', 0, NULL, "show aggregated counters" }, { "loop_ports", 'l', 0, NULL, "iterate through each port" }, { "reset_after_read", 'r', 0, NULL, "reset counters after read" }, @@ -410,6 +466,16 @@ int main(int argc, char **argv) all_ports_loop = 1; } + if (xmt_sl) { + xmt_sl_query(&portid, port, mask); + exit(0); + } + + if (rcv_sl) { + rcv_sl_query(&portid, port, mask); + exit(0); + } + if (all_ports_loop || (loop_ports && (all_ports || port == ALL_PORTS))) { if (smp_query(data, &portid, IB_ATTR_NODE_INFO, 0, 0) < 0) IBERROR("smp query nodeinfo failed"); From ogerlitz at Voltaire.com Sun Mar 1 04:59:35 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Sun, 01 Mar 2009 14:59:35 +0200 Subject: [ofa-general] [PATCH 1/2] libibmad: add PortXmtDataSL / PortRcvDataSL support In-Reply-To: <20090301081617.GJ3936@sashak.voltaire.com> References: <3B25B2D61996446F88703F647919FC4E@amr.corp.intel.com> <49AA3E52.30804@Voltaire.com> <20090301081617.GJ3936@sashak.voltaire.com> Message-ID: <49AA86B7.8050006@Voltaire.com> Sasha Khapyorsky wrote: > Also it would be nice (when possible) to add new field entries at end of > ib_mad_f array - in this way libibmad still be backward compatible. done in V2 Or. From ogerlitz at Voltaire.com Sun Mar 1 05:00:37 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Sun, 01 Mar 2009 15:00:37 +0200 Subject: [ofa-general] Re: [PATCH 2/2] perfquery: add PortXmtDataSL/PortRcvDataSL read and reset In-Reply-To: <20090301005952.GE3936@sashak.voltaire.com> References: <20090301005952.GE3936@sashak.voltaire.com> Message-ID: <49AA86F5.9070002@Voltaire.com> Sasha Khapyorsky wrote: > You can mask it by passing 's' as part of exclude string to > ibdiag_process_opts(). Or just find another, "free" latter for your option. I choose the second way, thanks Or. From amirv at mellanox.co.il Sun Mar 1 06:10:54 2009 From: amirv at mellanox.co.il (Amir Vadai) Date: Sun, 01 Mar 2009 16:10:54 +0200 Subject: [ofa-general] Measuring SDP throughput In-Reply-To: <49A878EE.70908@oracle.com> References: <49A878EE.70908@oracle.com> Message-ID: <49AA976E.80909@mellanox.co.il> Hi, I don't know of such a tool - except of course IB analyzer. I hope to add in the next release statistics in /proc for sdp, and you could use it than. - Amir On 02/28/2009 01:36 AM, Sumeet Lahorani wrote: > > Hi, > > Is there a tool to observe the SDP throughput while a workload is in > progress? I'm not looking for a tool such as qperf which generates > it's own workload. > > We have voltaire switches which give us overall throughput numbers > through the PortCounters.csv file but these are not limited to just > the SDP traffic. > > - Sumeet > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general From jackm at dev.mellanox.co.il Sun Mar 1 07:44:35 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Sun, 1 Mar 2009 17:44:35 +0200 Subject: ***SPAM*** Re: [ofa-general] ***SPAM*** Unable to get IPoIB working In-Reply-To: References: Message-ID: <200903011744.35202.jackm@dev.mellanox.co.il> On Monday 26 January 2009 22:44, Chuck Hartley wrote: > Is there some IPoIB debug I can turn on somehow? > On each of the hosts, you can do the following: in file /etc/modprobe.conf add the following line: options ib_ipoib debug_level=15 Then, restart the infiniband driver on both hosts: pkill opensm /etc/init.d/openibd restart You will see ipoib debugging output in file /var/log/messages (You should also probably remove (i.e., comment out) from file /etc/modprobe.conf any lines of the form "alias ib0 ib_ipoib" and "alias ib1 ib_ipoib", or the kernel will try to restart ipoib whenever you try to unload the driver -- not sure about this though for Fedora 9). - Jack From sashak at voltaire.com Sun Mar 1 13:33:20 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sun, 1 Mar 2009 23:33:20 +0200 Subject: [ofa-general] Re: [PATCH] opensm: Implement weighted routing In-Reply-To: <20090225210932.GA6098@sgi.com> References: <20090225210932.GA6098@sgi.com> Message-ID: <20090301204404.GN3936@sashak.voltaire.com> Hi Dale, On 15:09 Wed 25 Feb , Dale Purdy wrote: > > Implement a weighted routing scheme for fine tuning the lid matrix for > routing engines that use the lid matrix. An optional file containing > a switch_guid port and weighing factor combination per line can be > supplied to override a default hop weight factor of 1 for each switch > output port in computing the lid matrix. This allows one to alter the > min hop paths for things like routes to I/O. I'm fine with this patch. I guess that you are using this somehow. Could you provide usage example? I think it will be helpful for others to have it in opensm/doc/current-routing.txt or as separate file under opensm/doc. Thanks. > > Signed-off-by: Dale Purdy Applied. Thanks. Sasha From hal.rosenstock at gmail.com Sun Mar 1 17:23:04 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Sun, 1 Mar 2009 20:23:04 -0500 Subject: [ofa-general] [PATCH] IB/core: ib_post_send_mad() returns zero but doesn't generate send completion In-Reply-To: <1235771157.3948.233.camel@chromite.mv.qlogic.com> References: <1235771157.3948.233.camel@chromite.mv.qlogic.com> Message-ID: On Fri, Feb 27, 2009 at 4:45 PM, Ralph Campbell wrote: Good catches. > If ib_post_send_mad() returns zero, it guarantees that there will be > a callback to the send_buf->mad_agent->send_handler() so that the > sender can call ib_free_send_mad(). Otherwise, the ib_mad_send_buf > will be leaked and the mad_agent reference count will never go to zero > and the IB device module cannot be unloaded. > The above can happen without this patch if process_mad() returns > (IB_MAD_RESULT_SUCCESS | IB_MAD_RESULT_CONSUMED). This looks right although I'm not sure why this wasn't seen a long time ago. > If process_mad() returns IB_MAD_RESULT_SUCCESS and there is no agent > registered to receive the mad being sent, handle_outgoing_dr_smp() > returns zero which causes a MAD packet which is at the end of the > directed route to be incorrectly sent on the wire but doesn't cause > a hang since the HCA generates a send completion. This also looks right. Does the packet really get out on the IB wire ? Was that with a Mellanox HCA ? I don't recall ever seeing these extra packets but maybe they end up swallowed by certain HCAs. Unfortunately, I have no current way of regressing these changes. I'm presuming you've done this with Mellanox HCAs and with OpenSM running. One other question is whether this should be separated into two increments. -- Hal > Signed-off-by: Ralph Campbell > > diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c > index dbcd285..62a99dc 100644 > --- a/drivers/infiniband/core/mad.c > +++ b/drivers/infiniband/core/mad.c > @@ -742,9 +742,7 @@ static int handle_outgoing_dr_smp(struct ib_mad_agent_private *mad_agent_priv, >                break; >        case IB_MAD_RESULT_SUCCESS | IB_MAD_RESULT_CONSUMED: >                kmem_cache_free(ib_mad_cache, mad_priv); > -               kfree(local); > -               ret = 1; > -               goto out; > +               break; >        case IB_MAD_RESULT_SUCCESS: >                /* Treat like an incoming receive MAD */ >                port_priv = ib_get_mad_port(mad_agent_priv->agent.device, > @@ -755,10 +753,12 @@ static int handle_outgoing_dr_smp(struct ib_mad_agent_private *mad_agent_priv, >                                                        &mad_priv->mad.mad); >                } >                if (!port_priv || !recv_mad_agent) { > +                       /* > +                        * No receiving agent so drop packet and > +                        * generate send completion. > +                        */ >                        kmem_cache_free(ib_mad_cache, mad_priv); > -                       kfree(local); > -                       ret = 0; > -                       goto out; > +                       break; >                } >                local->mad_priv = mad_priv; >                local->recv_mad_agent = recv_mad_agent; > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hal.rosenstock at gmail.com Sun Mar 1 17:23:59 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Sun, 1 Mar 2009 20:23:59 -0500 Subject: ***SPAM*** Re: [ofa-general] [PATCH 11/10] libibmad:infiniband-diags: deprecate madrpc_set_[retries|timeout] WAS: [PATCH 1/10] libibmad: Clean up "new" interface In-Reply-To: <20090301072622.GG3936@sashak.voltaire.com> References: <20090219190525.322681b8.weiny2@llnl.gov> <20090220143402.c3b23b0a.weiny2@llnl.gov> <20090301072622.GG3936@sashak.voltaire.com> Message-ID: On Sun, Mar 1, 2009 at 2:26 AM, Sasha Khapyorsky wrote: [snip...] > Now you need to duplicate this single call over all tools. For me it > looks like an overkill. See below. > Probably it would be simpler to just read global > ibd_timeout variable in rpc.c? [snip...] > Same with retries - it is hard for me to believe that any multithreaded > application will try to setup different retry values per port, for > different threads, "on the fly".... For timeouts and retries, per port makes sense to me as ports may be connected to different fabrics with different timeouts. Also, it may depend on the operation (e.g. restricted to a single subnet or across multiple subnets). > (rpc.c with all its limited functionality will not be sufficient for such flexibility > level anyway :)). This would be an implementation issue to fix if the more flexible timeout/retries is to be implemented. -- Hal From hal.rosenstock at gmail.com Sun Mar 1 17:24:38 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Sun, 1 Mar 2009 20:24:38 -0500 Subject: ***SPAM*** Re: [ofa-general] [PATCH] opensm: PortInfo requests for discovered switches In-Reply-To: <20090228173635.GQ7462@sashak.voltaire.com> References: <20090228173635.GQ7462@sashak.voltaire.com> Message-ID: On Sat, Feb 28, 2009 at 12:36 PM, Sasha Khapyorsky wrote: > > Request PortInfo for all switch ports right on first NodeInfo > receiving and don't wait for SwitchInfo request results. This will > simplify a subnet discovery flow and speed it up. > Remove switch->discovery_count which is not needed anymore. > > Signed-off-by: Sasha Khapyorsky > --- >  opensm/include/opensm/osm_switch.h |    6 --- >  opensm/opensm/osm_node_info_rcv.c  |   83 ++++++++++++++---------------------- >  opensm/opensm/osm_perfmgr.c        |    1 - >  opensm/opensm/osm_state_mgr.c      |    1 - >  opensm/opensm/osm_sw_info_rcv.c    |   71 ------------------------------ >  5 files changed, 32 insertions(+), 130 deletions(-) > > diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h > index 6279727..3e3626b 100644 > --- a/opensm/include/opensm/osm_switch.h > +++ b/opensm/include/opensm/osm_switch.h > @@ -103,7 +103,6 @@ typedef struct osm_switch { >        uint8_t *lft; >        uint8_t *new_lft; >        osm_mcast_tbl_t mcast_tbl; > -       uint32_t discovery_count; >        unsigned endport_links; >        unsigned need_update; >        void *priv; > @@ -145,11 +144,6 @@ typedef struct osm_switch { >  *      mcast_tbl >  *              Multicast forwarding table for this switch. >  * > -*      discovery_count > -*              The number of times this switch has been discovered > -*              during the current fabric sweep.  This number is reset > -*              to zero at the start of a sweep. > -* >  *      need_update >  *              When set indicates that switch was probably reset, so >  *              fwd tables and rest cached data should be flushed > diff --git a/opensm/opensm/osm_node_info_rcv.c b/opensm/opensm/osm_node_info_rcv.c > index ac86b9a..e40fc82 100644 > --- a/opensm/opensm/osm_node_info_rcv.c > +++ b/opensm/opensm/osm_node_info_rcv.c > @@ -244,51 +244,43 @@ _exit: >  } > >  /********************************************************************** > - The plock must be held before calling this function. >  **********************************************************************/ > -static void > -__osm_ni_rcv_process_new_node(IN osm_sm_t * sm, > -                             IN osm_node_t * const p_node, > -                             IN const osm_madw_t * const p_madw) > +static void ni_rcv_get_port_info(IN osm_sm_t * sm, IN osm_node_t * node, > +                                IN const osm_madw_t * madw) >  { > -       ib_api_status_t status = IB_SUCCESS; >        osm_madw_context_t context; > -       osm_physp_t *p_physp; > -       ib_node_info_t *p_ni; > -       ib_smp_t *p_smp; > -       uint8_t port_num; > +       osm_physp_t *physp; > +       ib_node_info_t *ni; > +       unsigned port, num_ports; > +       ib_api_status_t status; > > -       OSM_LOG_ENTER(sm->p_log); > +       ni = ib_smp_get_payload_ptr(osm_madw_get_smp_ptr(madw)); > > -       p_smp = osm_madw_get_smp_ptr(p_madw); > -       p_ni = (ib_node_info_t *) ib_smp_get_payload_ptr(p_smp); > -       port_num = ib_node_info_get_local_port_num(p_ni); > +       if (ni->node_type == IB_NODE_TYPE_SWITCH) { > +               port = 0; > +               num_ports = osm_node_get_num_physp(node); > +       } else { > +               port = ib_node_info_get_local_port_num(ni); > +               num_ports = port + 1; The else clause determination of num_ports doesn't look right to me. Shouldn't it be num_ports = ni->num_ports ? -- Hal [snip...] From sashak at voltaire.com Sun Mar 1 23:19:58 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Mar 2009 09:19:58 +0200 Subject: [ofa-general] [PATCH] opensm: PortInfo requests for discovered switches In-Reply-To: References: <20090228173635.GQ7462@sashak.voltaire.com> Message-ID: <20090302071958.GO3936@sashak.voltaire.com> On 20:24 Sun 01 Mar , Hal Rosenstock wrote: > > + ?? ?? ?? if (ni->node_type == IB_NODE_TYPE_SWITCH) { > > + ?? ?? ?? ?? ?? ?? ?? port = 0; > > + ?? ?? ?? ?? ?? ?? ?? num_ports = osm_node_get_num_physp(node); > > + ?? ?? ?? } else { > > + ?? ?? ?? ?? ?? ?? ?? port = ib_node_info_get_local_port_num(ni); > > + ?? ?? ?? ?? ?? ?? ?? num_ports = port + 1; > > The else clause determination of num_ports doesn't look right to me. > Shouldn't it be > num_ports = ni->num_ports ? No, it's non-switch case where we are querying a single port. Sasha From sashak at voltaire.com Sun Mar 1 23:33:35 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Mar 2009 09:33:35 +0200 Subject: [ofa-general] Re: [PATCH] ibsim/sim.h: Better portinfo alignment in Port struct In-Reply-To: <20090220173711.GA3024@comcast.net> References: <20090220173711.GA3024@comcast.net> Message-ID: <20090302073335.GP3936@sashak.voltaire.com> On 12:37 Fri 20 Feb , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Sun Mar 1 23:35:39 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Mar 2009 09:35:39 +0200 Subject: [ofa-general] Re: [PATCH] ibsim: Handle sim_init_net errors better In-Reply-To: <20090220203336.GA3874@comcast.net> References: <20090220203336.GA3874@comcast.net> Message-ID: <20090302073539.GQ3936@sashak.voltaire.com> On 15:33 Fri 20 Feb , Hal Rosenstock wrote: > > Use define rather than constant > Also, cosmetic formatting and fixed some typos > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From nicolas.morey-chaisemartin at ext.bull.net Mon Mar 2 00:06:08 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Mon, 02 Mar 2009 09:06:08 +0100 Subject: [ofa-general] [RFC] Fat-Tree upgrades In-Reply-To: <498FDDD5.1090204@ext.bull.net> References: <498FDDD5.1090204@ext.bull.net> Message-ID: <49AB9370.8050102@ext.bull.net> Nicolas Morey Chaisemartin wrote: > Another thing we have developped here is to balance more secondary path. > In the current algorithm, secondary down path (going_down_by_going up) > are created in port_group order. > This means that if the primary path didn't reach all the network > (because a switch is broken for examples), all the routes missing will > be created through the first port group. Which unbalance the network > load a lot. > To solve this, we create the secondary path by port group load. > The previous patch has made us increment the port/portgroup counters > when secondary routes towards HCA are created, therefore these counters > are significant even when creating secondary routes. > What our patch does is at the beginning of the function sort all the > port group from lowest load to highest. Pick the first one for the > primary path, and try secondary path from the 2nd to the last. > Once again this seems to have no effect on regular topology but it made > a real impact on our failover tests. > > After some work on this, I managed to achieve even better results. When both links are equally loaded, I look which one of the remote switch as the least loaded down route. This way when all the switch are equivalent to go up, we choose the best one to go down at the next step. I'm still running some tests but early results show much more balanced results than the previous ones. I'd still like to hear comments about this approach and the previous one. It would be a pity not to share the work we have done on the algorithm. Nicolas From sashak at voltaire.com Mon Mar 2 00:38:04 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Mar 2009 10:38:04 +0200 Subject: [ofa-general] Re: [PATCH v2 0/4] generic perf query/reset api plus PortXmtDataSL/PortRcvDataSL impl In-Reply-To: References: Message-ID: <20090302083756.GT3936@sashak.voltaire.com> On 14:37 Sun 01 Mar , Or Gerlitz wrote: > Following the comments from the initial version, this patch series first > exports a generic libibmad api to issue query and reset on performance > counters and then implements PortXmtDataSL and PortRcvDataSL. Applied. Thanks. I merged patches 1 and 2 in order to not break management tree build. Will note ask question about patch 4 there. Sasha From sashak at voltaire.com Mon Mar 2 00:40:27 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Mar 2009 10:40:27 +0200 Subject: [ofa-general] Re: [PATCH v2 4/4] perfquery PortXmtDataSL/PortRcvDataSL support In-Reply-To: References: Message-ID: <20090302084027.GU3936@sashak.voltaire.com> On 14:41 Sun 01 Mar , Or Gerlitz wrote: > @@ -353,6 +407,8 @@ int main(int argc, char **argv) > > const struct ibdiag_opt opts[] = { > { "extended", 'x', 0, NULL, "show extended port counters" }, > + { "xmtsl", 'X', 0, NULL, "show Xmt SL port counters" }, > + { "rcvsl", 'S', 0, NULL, "show Rcv SL port counters" }, > { "all_ports", 'a', 0, NULL, "show aggregated counters" }, > { "loop_ports", 'l', 0, NULL, "iterate through each port" }, > { "reset_after_read", 'r', 0, NULL, "reset counters after read" }, > @@ -410,6 +466,16 @@ int main(int argc, char **argv) > all_ports_loop = 1; > } > > + if (xmt_sl) { > + xmt_sl_query(&portid, port, mask); > + exit(0); > + } > + > + if (rcv_sl) { > + rcv_sl_query(&portid, port, mask); > + exit(0); > + } Would options '-S -X' useful together? May it work with '-a' (all ports)? Sasha From sashak at voltaire.com Mon Mar 2 00:49:34 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Mar 2009 10:49:34 +0200 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] opensm/console: Enhance perfmgr print_counters for better nodenames In-Reply-To: <20090227211033.GC16941@obsidianresearch.com> References: <20090219130653.GA29318@comcast.net> <20090226061551.GQ11192@sashak.voltaire.com> <20090227092405.GD7462@sashak.voltaire.com> <20090227211033.GC16941@obsidianresearch.com> Message-ID: <20090302084927.GV3936@sashak.voltaire.com> On 14:10 Fri 27 Feb , Jason Gunthorpe wrote: > > Is this really worth doing? // is included in C99 and many other > C99isms are already used in the source (well, until Sean removes > them.. :) And seems it is only C99ism supported by VC :) Sasha From ogerlitz at voltaire.com Mon Mar 2 00:54:08 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Mon, 02 Mar 2009 10:54:08 +0200 Subject: [ofa-general] Re: [PATCH v2 4/4] perfquery PortXmtDataSL/PortRcvDataSL support In-Reply-To: <20090302084027.GU3936@sashak.voltaire.com> References: <20090302084027.GU3936@sashak.voltaire.com> Message-ID: <49AB9EB0.8020006@voltaire.com> Sasha Khapyorsky wrote: > Would options '-S -X' useful together? May it work with '-a' (all ports)? > I'm not sure how much its useful, I also don't have hardware that supports both queries on the same port at the same time. I have tested the -a option and it works fine (sets port to 0xff, etc). Or. From ogerlitz at voltaire.com Mon Mar 2 01:35:54 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Mon, 2 Mar 2009 11:35:54 +0200 (IST) Subject: [ofa-general] [PATCH] infiniband-diags: update configure.in check for libibmad API In-Reply-To: References: Message-ID: update infiniband-diags/configure.in check for libibmad API Signed-off-by: Or Gerlitz Index: management/infiniband-diags/configure.in =================================================================== --- management.orig/infiniband-diags/configure.in +++ management/infiniband-diags/configure.in @@ -36,8 +36,8 @@ AC_CHECK_LIB(ibumad, umad_init, [], AC_MSG_ERROR([umad_init() not found. diags require libibumad.])) AC_CHECK_LIB(ibmad, mad_dump_int, [], AC_MSG_ERROR([mad_dump_int() not found. diags require libibmad.])) -AC_CHECK_LIB(ibmad, port_performance_ext_query, [], - AC_MSG_ERROR([port_performance_ext_query() not found. diags require more recent libibmad.])) +AC_CHECK_LIB(ibmad, pma_query_via, [], + AC_MSG_ERROR([pma_query_via() not found. diags require more recent libibmad.])) AC_CHECK_LIB(osmcomp, cl_thread_init, [], AC_MSG_ERROR([cl_thread_init() not found. diags require libosmcomp.])) fi From sashak at voltaire.com Mon Mar 2 02:06:07 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 2 Mar 2009 12:06:07 +0200 Subject: [ofa-general] Re: [PATCH] infiniband-diags: update configure.in check for libibmad API In-Reply-To: References: Message-ID: <20090302100607.GW3936@sashak.voltaire.com> On 11:35 Mon 02 Mar , Or Gerlitz wrote: > update infiniband-diags/configure.in check for libibmad API > > Signed-off-by: Or Gerlitz Applied. Thanks. Sasha From vlad at lists.openfabrics.org Mon Mar 2 03:40:32 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Mon, 2 Mar 2009 03:40:32 -0800 (PST) Subject: [ofa-general] ofa_1_4_kernel 20090302-0200 daily build status Message-ID: <20090302114032.AB95CE60E76@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: Build failed on x86_64 with linux-2.6.18-93.el5 Log: /home/vlad/tmp/ofa_1_4_kernel-20090302-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/ulp/iser/iser_memory.c:253: warning: assignment makes pointer from integer without a cast /home/vlad/tmp/ofa_1_4_kernel-20090302-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/ulp/iser/iser_memory.c: In function 'iser_data_buf_aligned_len': /home/vlad/tmp/ofa_1_4_kernel-20090302-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/ulp/iser/iser_memory.c:311: warning: passing argument 2 of 'ib_sg_dma_address' makes pointer from integer without a cast make[4]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090302-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/ulp/iser/iser_memory.o] Error 1 make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090302-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband/ulp/iser] Error 2 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090302-0200_linux-2.6.18-93.el5_x86_64_check/drivers/infiniband] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090302-0200_linux-2.6.18-93.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-93.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From hnrose at comcast.net Mon Mar 2 04:03:57 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Mon, 2 Mar 2009 07:03:57 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] opensm/ib_types.h: Add attribute ID for PortCountersExtended Message-ID: <20090302120357.GB6493@comcast.net> Also, add some PerfMgt ClassPortInfo CapabilityMask bit definitions Signed-off-by: Hal Rosenstock --- diff --git a/opensm/include/iba/ib_types.h b/opensm/include/iba/ib_types.h index 780246c..205d160 100644 --- a/opensm/include/iba/ib_types.h +++ b/opensm/include/iba/ib_types.h @@ -1029,6 +1029,18 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_PORT_CNTRS (CL_HTON16(0x0012)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_PORT_CNTRS_EXT +* NAME +* IB_MAD_ATTR_PORT_CNTRS_EXT +* +* DESCRIPTION +* PortCountersExtended attribute (16.1.4) +* +* SOURCE +*/ +#define IB_MAD_ATTR_PORT_CNTRS_EXT (CL_HTON16(0x001D)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_GUID_INFO * NAME * IB_MAD_ATTR_GUID_INFO @@ -3244,7 +3256,6 @@ ib_path_rec_hop_limit(IN const ib_path_rec_t * const p_rec) * * DESCRIPTION * IBA defined ClassPortInfo attribute (13.4.8.1) -* route between two end-points on a subnet. * * SYNOPSIS */ @@ -3327,6 +3338,10 @@ typedef struct _ib_class_port_info { * *********/ +#define IB_PM_ALL_PORT_SELECT (CL_HTON16(((uint16_t)1)<<8)) +#define IB_PM_EXT_WIDTH_SUPPORTED (CL_HTON16(((uint16_t)1)<<9)) +#define IB_PM_EXT_WIDTH_NOIETF_SUP (CL_HTON16(((uint16_t)1)<<10)) + /****f* IBA Base: Types/ib_class_set_resp_time_val * NAME * ib_class_set_resp_time_val From hnrose at comcast.net Mon Mar 2 04:03:06 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Mon, 2 Mar 2009 07:03:06 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] opensm: Return error status when cl_disp_register fails Message-ID: <20090302120306.GA6493@comcast.net> Also, in perfmgr, return error on db_construct failure Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_perfmgr.c b/opensm/opensm/osm_perfmgr.c index 58b5dc2..43278ae 100644 --- a/opensm/opensm/osm_perfmgr.c +++ b/opensm/opensm/osm_perfmgr.c @@ -1215,7 +1215,7 @@ ib_api_status_t osm_perfmgr_init(osm_perfmgr_t * const pm, osm_opensm_t *osm, const osm_subn_opt_t * const p_opt) { - ib_api_status_t status = IB_SUCCESS; + ib_api_status_t status; OSM_LOG_ENTER(&osm->log); @@ -1242,6 +1242,7 @@ osm_perfmgr_init(osm_perfmgr_t * const pm, osm_opensm_t *osm, if (status != IB_SUCCESS) goto Exit; + status = IB_INSUFFICIENT_RESOURCES; pm->db = perfmgr_db_construct(pm); if (!pm->db) { pm->state = PERFMGR_STATE_NO_DB; @@ -1259,6 +1260,7 @@ osm_perfmgr_init(osm_perfmgr_t * const pm, osm_opensm_t *osm, cl_timer_start(&pm->sweep_timer, pm->sweep_time_s * 1000); + status = IB_SUCCESS; Exit: OSM_LOG_EXIT(pm->log); return (status); diff --git a/opensm/opensm/osm_sa.c b/opensm/opensm/osm_sa.c index 416d44a..50c4d45 100644 --- a/opensm/opensm/osm_sa.c +++ b/opensm/opensm/osm_sa.c @@ -163,7 +163,7 @@ osm_sa_init(IN osm_sm_t * const p_sm, IN osm_stats_t * const p_stats, IN cl_dispatcher_t * const p_disp, IN cl_plock_t * const p_lock) { - ib_api_status_t status = IB_SUCCESS; + ib_api_status_t status; OSM_LOG_ENTER(p_log); @@ -189,6 +189,7 @@ osm_sa_init(IN osm_sm_t * const p_sm, if (status != IB_SUCCESS) goto Exit; + status = IB_INSUFFICIENT_RESOURCES; p_sa->cpi_disp_h = cl_disp_register(p_disp, OSM_MSG_MAD_CLASS_PORT_INFO, osm_cpi_rcv_process, p_sa); if (p_sa->cpi_disp_h == CL_DISP_INVALID_HANDLE) @@ -287,6 +288,7 @@ osm_sa_init(IN osm_sm_t * const p_sm, if (p_sa->mft_disp_h == CL_DISP_INVALID_HANDLE) goto Exit; + status = IB_SUCCESS; Exit: OSM_LOG_EXIT(p_log); return (status); diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c index d1d8863..60dcdbb 100644 --- a/opensm/opensm/osm_sm.c +++ b/opensm/opensm/osm_sm.c @@ -254,7 +254,7 @@ osm_sm_init(IN osm_sm_t * const p_sm, IN osm_stats_t * const p_stats, IN cl_dispatcher_t * const p_disp, IN cl_plock_t * const p_lock) { - ib_api_status_t status = IB_SUCCESS; + ib_api_status_t status; OSM_LOG_ENTER(p_log); @@ -319,6 +319,7 @@ osm_sm_init(IN osm_sm_t * const p_sm, if (status != IB_SUCCESS) goto Exit; + status = IB_INSUFFICIENT_RESOURCES; p_sm->sweep_fail_disp_h = cl_disp_register(p_disp, OSM_MSG_LIGHT_SWEEP_FAIL, sweep_fail_process, p_sm); From klakshman03 at hotmail.com Mon Mar 2 06:38:54 2009 From: klakshman03 at hotmail.com (lakshmana swamy) Date: Mon, 2 Mar 2009 20:08:54 +0530 Subject: [ofa-general] ***SPAM*** RE: Problem in IB network without Switch In-Reply-To: <200902261849.40448.jackm@dev.mellanox.co.il> References: <829ded920902260031r6f8b973t9f2e536864e25c85@mail.gmail.com> <200902261335.59927.jackm@dev.mellanox.co.il> <200902261849.40448.jackm@dev.mellanox.co.il> Message-ID: HI Jack I have updated the firmware of HCA in both the machines, but the status remains same. Please have a look at the following outputs. What may be the problem ? [root at locluster ~]# ibstat CA 'mthca0' CA type: MT23108 Number of ports: 2 Firmware version: 3.5.917 Hardware version: a1 Node GUID: 0x0002c90108cd13c0 System image GUID: 0x0002c90108cd13c0 Port 1: State: Down Physical state: Polling Rate: 2 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02510a6a Port GUID: 0x0002c90108cd13c1 Port 2: State: Down Physical state: Polling Rate: 2 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02510a68 Port GUID: 0x0002c90108cd13c2 [root at locluster ~]# [root at locluster1 ~]# ibstat CA 'mthca0' CA type: MT23108 Number of ports: 2 Firmware version: 3.5.917 Hardware version: a1 Node GUID: 0x0002c9020000114c System image GUID: 0x0002c9020000114f Port 1: State: Down Physical state: Polling Rate: 2 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02510a68 Port GUID: 0x0002c9020000114d Port 2: State: Down Physical state: Polling Rate: 2 Base lid: 0 LMC: 0 SM lid: 0 Capability mask: 0x02510a68 Port GUID: 0x0002c9020000114e [root at locluster1 ~]# ThankQ laxman From: klakshman03 at hotmail.com To: jackm at dev.mellanox.co.il CC: keshetti.mahesh at gmail.com; general at lists.openfabrics.org Subject: RE: Problem in IB network without Switch Date: Fri, 27 Feb 2009 09:56:38 +0530 ThanQ Jack I will Update the firmware and let you know the status laxman > From: jackm at dev.mellanox.co.il > To: klakshman03 at hotmail.com > Subject: Re: Problem in IB network without Switch > Date: Thu, 26 Feb 2009 18:49:40 +0200 > CC: keshetti.mahesh at gmail.com; general at lists.openfabrics.org > > You are running VERY old firmware (from 2004), and moreover, on one host > you have 3.0.0, and on the other 3.1.0. > > You need to upgrade your firmware. > Contact your Mellanox FAE (support engineer) for instructions. > > - Jack > > > Hi Jack, > > > > Please find the output of ibstat on both the nodes, . > > > > [root at mattool ~]# /opt/ofed/extras/hca_self_test.ofed > > HCA Firmware Check ..................... FAIL > > REASON: mismatch HCA #0 firmware detected (found v, need v3.5.917) > > Host Driver Initialization ............. PASS > > > > [root at mattool ~]# > > > > ************ IBSTAT output ****************** > > > > > > [root at mattool ~]# ibstat > > CA 'mthca0' > > CA type: MT23108 > > Number of ports: 2 > > Firmware version: 3.1.0 > > > [root at compute-0-0 ~]# ibstat > > CA 'mthca0' > > CA type: MT23108 > > Number of ports: 2 > > Firmware version: 3.0.0 Get a view of the world through MSN Video. Some things just cannot be left unseen. Try it! _________________________________________________________________ Express your true emotions! Send a virtual kiss! Do all these cool stuff & more with Windows Live Messenger. Download here http://www.windowslive.com/Desktop/Messenger -------------- next part -------------- An HTML attachment was scrubbed... URL: From jackm at dev.mellanox.co.il Mon Mar 2 06:46:46 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 2 Mar 2009 16:46:46 +0200 Subject: [ofa-general] Re: Problem in IB network without Switch In-Reply-To: References: <829ded920902260031r6f8b973t9f2e536864e25c85@mail.gmail.com> <200902261849.40448.jackm@dev.mellanox.co.il> Message-ID: <200903021646.47008.jackm@dev.mellanox.co.il> On Monday 02 March 2009 16:38, lakshmana swamy wrote: > > HI Jack > > I have updated the firmware of HCA in both the machines, but the status remains same. > Please have a look at the following outputs. > > What may be the problem ? > Your physical connection is bad. Check your cables. Try connecting Port 1 to Port 2 of the same HCA directly, and see if you still get state: Down. (you should see state: init before running opensm). - Jack > [root at locluster ~]# ibstat > CA 'mthca0' > CA type: MT23108 > Number of ports: 2 > Firmware version: 3.5.917 > Hardware version: a1 > Node GUID: 0x0002c90108cd13c0 > System image GUID: 0x0002c90108cd13c0 > Port 1: > State: Down > Physical state: Polling > Rate: 2 > Base lid: 0 > LMC: 0 > SM lid: 0 > Capability mask: 0x02510a6a > Port GUID: 0x0002c90108cd13c1 > Port 2: > State: Down > Physical state: Polling > Rate: 2 > Base lid: 0 > LMC: 0 > SM lid: 0 > Capability mask: 0x02510a68 > Port GUID: 0x0002c90108cd13c2 > [root at locluster ~]# > > [root at locluster1 ~]# ibstat > CA 'mthca0' > CA type: MT23108 > Number of ports: 2 > Firmware version: 3.5.917 > Hardware version: a1 > Node GUID: 0x0002c9020000114c > System image GUID: 0x0002c9020000114f > Port 1: > State: Down > Physical state: Polling > Rate: 2 > Base lid: 0 > LMC: 0 > SM lid: 0 > Capability mask: 0x02510a68 > Port GUID: 0x0002c9020000114d > Port 2: > State: Down > Physical state: Polling > Rate: 2 > Base lid: 0 > LMC: 0 > SM lid: 0 > Capability mask: 0x02510a68 > Port GUID: 0x0002c9020000114e > [root at locluster1 ~]# > > > > ThankQ > > laxman > > > From: klakshman03 at hotmail.com > To: jackm at dev.mellanox.co.il > CC: keshetti.mahesh at gmail.com; general at lists.openfabrics.org > Subject: RE: Problem in IB network without Switch > Date: Fri, 27 Feb 2009 09:56:38 +0530 > > > > > > > > > ThanQ Jack > > I will Update the firmware and let you know the status > > laxman > > > From: jackm at dev.mellanox.co.il > > To: klakshman03 at hotmail.com > > Subject: Re: Problem in IB network without Switch > > Date: Thu, 26 Feb 2009 18:49:40 +0200 > > CC: keshetti.mahesh at gmail.com; general at lists.openfabrics.org > > > > You are running VERY old firmware (from 2004), and moreover, on one host > > you have 3.0.0, and on the other 3.1.0. > > > > You need to upgrade your firmware. > > Contact your Mellanox FAE (support engineer) for instructions. > > > > - Jack > > > > > Hi Jack, > > > > > > Please find the output of ibstat on both the nodes, . > > > > > > [root at mattool ~]# /opt/ofed/extras/hca_self_test.ofed > > > HCA Firmware Check ..................... FAIL > > > REASON: mismatch HCA #0 firmware detected (found v, need v3.5.917) > > > Host Driver Initialization ............. PASS > > > > > > [root at mattool ~]# > > > > > > ************ IBSTAT output ****************** > > > > > > > > > [root at mattool ~]# ibstat > > > CA 'mthca0' > > > CA type: MT23108 > > > Number of ports: 2 > > > Firmware version: 3.1.0 > > > > > [root at compute-0-0 ~]# ibstat > > > CA 'mthca0' > > > CA type: MT23108 > > > Number of ports: 2 > > > Firmware version: 3.0.0 > > Get a view of the world through MSN Video. Some things just cannot be left unseen. Try it! > _________________________________________________________________ > Express your true emotions! Send a virtual kiss! Do all these cool stuff & more with Windows Live Messenger. Download here > http://www.windowslive.com/Desktop/Messenger From klakshman03 at hotmail.com Mon Mar 2 07:07:31 2009 From: klakshman03 at hotmail.com (lakshmana swamy) Date: Mon, 2 Mar 2009 20:37:31 +0530 Subject: [ofa-general] ***SPAM*** RE: Problem in IB network without Switch In-Reply-To: <200903021646.47008.jackm@dev.mellanox.co.il> References: <829ded920902260031r6f8b973t9f2e536864e25c85@mail.gmail.com> <200902261849.40448.jackm@dev.mellanox.co.il> <200903021646.47008.jackm@dev.mellanox.co.il> Message-ID: Hi Jack, Yes, I connected the cable between two ports of same HCA. Without running opensmd. Now the State is " Initializing" I observed same state in other node also. If cable is bad, it should not show this state right... Thanks laxman > From: jackm at dev.mellanox.co.il > To: klakshman03 at hotmail.com > Subject: Re: Problem in IB network without Switch > Date: Mon, 2 Mar 2009 16:46:46 +0200 > CC: keshetti.mahesh at gmail.com; general at lists.openfabrics.org > > On Monday 02 March 2009 16:38, lakshmana swamy wrote: > > > > > HI Jack > > > > I have updated the firmware of HCA in both the machines, but the status remains same. > > Please have a look at the following outputs. > > > > What may be the problem ? > > > Your physical connection is bad. Check your cables. > > Try connecting Port 1 to Port 2 of the same HCA directly, and see if you still get state: Down. > (you should see state: init before running opensm). > > - Jack > > > [root at locluster ~]# ibstat > > CA 'mthca0' > > CA type: MT23108 > > Number of ports: 2 > > Firmware version: 3.5.917 > > Hardware version: a1 > > Node GUID: 0x0002c90108cd13c0 > > System image GUID: 0x0002c90108cd13c0 > > Port 1: > > State: Down > > Physical state: Polling > > Rate: 2 > > Base lid: 0 > > LMC: 0 > > SM lid: 0 > > Capability mask: 0x02510a6a > > Port GUID: 0x0002c90108cd13c1 > > Port 2: > > State: Down > > Physical state: Polling > > Rate: 2 > > Base lid: 0 > > LMC: 0 > > SM lid: 0 > > Capability mask: 0x02510a68 > > Port GUID: 0x0002c90108cd13c2 > > [root at locluster ~]# > > > > [root at locluster1 ~]# ibstat > > CA 'mthca0' > > CA type: MT23108 > > Number of ports: 2 > > Firmware version: 3.5.917 > > Hardware version: a1 > > Node GUID: 0x0002c9020000114c > > System image GUID: 0x0002c9020000114f > > Port 1: > > State: Down > > Physical state: Polling > > Rate: 2 > > Base lid: 0 > > LMC: 0 > > SM lid: 0 > > Capability mask: 0x02510a68 > > Port GUID: 0x0002c9020000114d > > Port 2: > > State: Down > > Physical state: Polling > > Rate: 2 > > Base lid: 0 > > LMC: 0 > > SM lid: 0 > > Capability mask: 0x02510a68 > > Port GUID: 0x0002c9020000114e > > [root at locluster1 ~]# > > > > > > > > ThankQ > > > > laxman > > > > > > From: klakshman03 at hotmail.com > > To: jackm at dev.mellanox.co.il > > CC: keshetti.mahesh at gmail.com; general at lists.openfabrics.org > > Subject: RE: Problem in IB network without Switch > > Date: Fri, 27 Feb 2009 09:56:38 +0530 > > > > > > > > > > > > > > > > > > ThanQ Jack > > > > I will Update the firmware and let you know the status > > > > laxman > > > > > From: jackm at dev.mellanox.co.il > > > To: klakshman03 at hotmail.com > > > Subject: Re: Problem in IB network without Switch > > > Date: Thu, 26 Feb 2009 18:49:40 +0200 > > > CC: keshetti.mahesh at gmail.com; general at lists.openfabrics.org > > > > > > You are running VERY old firmware (from 2004), and moreover, on one host > > > you have 3.0.0, and on the other 3.1.0. > > > > > > You need to upgrade your firmware. > > > Contact your Mellanox FAE (support engineer) for instructions. > > > > > > - Jack > > > > > > > Hi Jack, > > > > > > > > Please find the output of ibstat on both the nodes, . > > > > > > > > [root at mattool ~]# /opt/ofed/extras/hca_self_test.ofed > > > > HCA Firmware Check ..................... FAIL > > > > REASON: mismatch HCA #0 firmware detected (found v, need v3.5.917) > > > > Host Driver Initialization ............. PASS > > > > > > > > [root at mattool ~]# > > > > > > > > ************ IBSTAT output ****************** > > > > > > > > > > > > [root at mattool ~]# ibstat > > > > CA 'mthca0' > > > > CA type: MT23108 > > > > Number of ports: 2 > > > > Firmware version: 3.1.0 > > > > > > > [root at compute-0-0 ~]# ibstat > > > > CA 'mthca0' > > > > CA type: MT23108 > > > > Number of ports: 2 > > > > Firmware version: 3.0.0 > > > > Get a view of the world through MSN Video. Some things just cannot be left unseen. Try it! > > _________________________________________________________________ > > Express your true emotions! Send a virtual kiss! Do all these cool stuff & more with Windows Live Messenger. Download here > > http://www.windowslive.com/Desktop/Messenger _________________________________________________________________ Express your true emotions! Send a virtual kiss! Do all these cool stuff & more with Windows Live Messenger. Download here http://www.windowslive.com/Desktop/Messenger -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolas.morey-chaisemartin at ext.bull.net Mon Mar 2 07:16:04 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Mon, 02 Mar 2009 16:16:04 +0100 Subject: [ofa-general] [PATCH] opensm/console: Fixed osm_console poll to so it won't eat up all CPU when stdin is a pipe Message-ID: <49ABF834.90106@ext.bull.net> Signed-off-by: Nicolas Morey-Chaisemartin --- When opensm is called after a pipe to execute a command in the local console, it uses 100% CPU as osm_console poll returns as if there was something to read from stdin where there isn't. This patch fixes by closing the local console when the end of osm_console function has been reached, meaning poll returned a positive value but actually nothing to read. osm_console will return an error code which will unset the console_init_flag so it won't be reopened. For socket, the osm_console_close is called as it means the socket isn't in a proper state. opensm/include/opensm/osm_console.h | 2 +- opensm/opensm/main.c | 6 ++++-- opensm/opensm/osm_console.c | 19 +++++++++++++++---- 3 files changed, 20 insertions(+), 7 deletions(-) diff --git a/opensm/include/opensm/osm_console.h b/opensm/include/opensm/osm_console.h index 3ea8fa5..dcce1e4 100644 --- a/opensm/include/opensm/osm_console.h +++ b/opensm/include/opensm/osm_console.h @@ -46,6 +46,6 @@ BEGIN_C_DECLS // TODO replace p_osm -void osm_console(osm_opensm_t * p_osm); +int osm_console(osm_opensm_t * p_osm); END_C_DECLS #endif /* _OSM_CONSOLE_H_ */ diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c index 2271634..5a39f84 100644 --- a/opensm/opensm/main.c +++ b/opensm/opensm/main.c @@ -494,8 +494,10 @@ int osm_manager_loop(osm_subn_opt_t * p_opt, osm_opensm_t * p_osm) Sit here forever - dwell or do console i/o & cmds */ while (!osm_exit_flag) { - if (console_init_flag) - osm_console(p_osm); + if (console_init_flag){ + if(osm_console(p_osm)) + console_init_flag = 0; + } else cl_thread_suspend(10000); diff --git a/opensm/opensm/osm_console.c b/opensm/opensm/osm_console.c index 63c5ea8..b6a8689 100644 --- a/opensm/opensm/osm_console.c +++ b/opensm/opensm/osm_console.c @@ -1376,7 +1376,7 @@ static void parse_cmd_line(char *line, osm_opensm_t * p_osm) } } -void osm_console(osm_opensm_t * p_osm) +int osm_console(osm_opensm_t * p_osm) { struct pollfd pollfd[2]; char *p_line; @@ -1409,7 +1409,7 @@ void osm_console(osm_opensm_t * p_osm) } if (poll(fds, nfds, 1000) <= 0) - return; + return 0; #ifdef ENABLE_OSM_CONSOLE_SOCKET if (pollfd[0].revents & POLLIN) { @@ -1422,7 +1422,7 @@ void osm_console(osm_opensm_t * p_osm) "ERR 4B04: Failed to accept console socket: %s\n", strerror(errno)); p_oct->in_fd = -1; - return; + return 0; } if (inet_ntop (AF_INET, &sin.sin_addr, p_oct->client_ip, @@ -1444,7 +1444,7 @@ void osm_console(osm_opensm_t * p_osm) p_oct->client_hn, p_oct->client_ip); close(new_fd); } - return; + return 0; } #endif @@ -1462,5 +1462,16 @@ void osm_console(osm_opensm_t * p_osm) osm_console_exit(p_oct, p_log); if (p_line) free(p_line); + return 0; } + /* If we are here, there has been a problem with poll + * so we exit the console not to eat up all the CPU + * if it was a local console + * If it was a socket, we just close it. + */ + if( p_oct->socket < 0 ) + return -1; + + osm_console_exit(p_oct, p_log); + return 0; } -- 1.5.6.3 From hal.rosenstock at gmail.com Mon Mar 2 07:28:02 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Mon, 2 Mar 2009 10:28:02 -0500 Subject: ***SPAM*** Re: [ofa-general] [PATCH] opensm: PortInfo requests for discovered switches In-Reply-To: <20090302071958.GO3936@sashak.voltaire.com> References: <20090228173635.GQ7462@sashak.voltaire.com> <20090302071958.GO3936@sashak.voltaire.com> Message-ID: On Mon, Mar 2, 2009 at 2:19 AM, Sasha Khapyorsky wrote: > On 20:24 Sun 01 Mar     , Hal Rosenstock wrote: >> > + ?? ?? ?? if (ni->node_type == IB_NODE_TYPE_SWITCH) { >> > + ?? ?? ?? ?? ?? ?? ?? port = 0; >> > + ?? ?? ?? ?? ?? ?? ?? num_ports = osm_node_get_num_physp(node); >> > + ?? ?? ?? } else { >> > + ?? ?? ?? ?? ?? ?? ?? port = ib_node_info_get_local_port_num(ni); >> > + ?? ?? ?? ?? ?? ?? ?? num_ports = port + 1; >> >> The else clause determination of num_ports doesn't look right to me. >> Shouldn't it be >> num_ports = ni->num_ports ? > > No, it's non-switch case where we are querying a single port. Oh I see; it's the below: for (; port < num_ports; port++) { -- Hal > Sasha > From jeff.scott at openfabrics.org Mon Mar 2 09:17:51 2009 From: jeff.scott at openfabrics.org (Jeffrey Scott) Date: Mon, 02 Mar 2009 09:17:51 -0800 Subject: [ofa-general] Sonoma Early Bird registration deadline - March 6 Message-ID: <09B36F3D2CCA484486C7660CF4335464@Gaucho> The deadline for discounted 'Early Bird' registrations to the Sonoma Workshop has been extended to this Friday, March 6. The discounted rate is $495. You can register for the event and book a hotel room at http://www.acteva.com/booking.cfm?bevaid=173845. Ronald Luijten of the IBM Zurich Research Lab will kickoff the Sonoma Workshop with a keynote on the evening of Sunday, March 22. The keynote will examine the requirements of network transports and wire protocols for 40 and 100 gigabit data rates. The entire agenda for the March 22-25 event is packed with great content, including speakers from AMD, Cisco, Intel, General Motors, Goldman Sachs, Microsoft, NASA, Oracle, the Linux Foundation and more. We will also hear from Addison Snell, Vice President and General Manager of Tabor Research, who will discuss the future of low-latency fabrics for HPC and enterprise datacenters, including the projected adoption rate for these fabrics and the likely impact of these technologies in the years ahead. A preliminary agenda is attached. Membership in the Alliance is not necessary to attend the Sonoma Workshop. Please pass along event information to any colleagues, customers, partners or business associates who may be interested in attending. More details about the Sonoma Workshop are available at www.OpenFabrics.org . The OpenFabrics Alliance is looking forward to a great event. We hope to see everyone in Sonoma. OpenFabrics Alliance Marketing Working Group -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Sonoma 2009 preliminary agenda.pdf Type: application/pdf Size: 37259 bytes Desc: not available URL: From weiny2 at llnl.gov Mon Mar 2 09:54:59 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Mon, 2 Mar 2009 09:54:59 -0800 Subject: [ofa-general] Re: [PATCH 1/10] libibmad: Clean up "new" interface In-Reply-To: <20090301070013.GF3936@sashak.voltaire.com> References: <20090219190525.322681b8.weiny2@llnl.gov> <20090301070013.GF3936@sashak.voltaire.com> Message-ID: <20090302095459.7e6d4a34.weiny2@llnl.gov> On Sun, 1 Mar 2009 09:00:20 +0200 Sasha Khapyorsky wrote: > Hi Ira, > > On 19:05 Thu 19 Feb , Ira Weiny wrote: > > From 2774b4ab4608e25bdc365bca3a94c7d51ee19372 Mon Sep 17 00:00:00 2001 > > From: Ira Weiny > > Date: Wed, 18 Feb 2009 16:37:36 -0800 > > Subject: [PATCH] libibmad: Clean up "new" interface > > Please don't put email header into commit message body, it breaks tools > like 'git rebase' and similar. At least put '>' before first 'From ' > line. You never mentioned it before, I guess something has changed on my end, sorry. > > > > > type all "void *ibmad_port" and "void *srcport" with struct ibmad_port * > > Do you plan to expose 'struct ibmad_port' (I see later in patches that > it is going to some libibmad internal header file)? No, I don't plan on exposing the internals. We can keep it a generic type as far as the user is concerned. The move to an internal header was to be able to use it across other internal *.c files. > > > Create new mad_rpc_portid(struct ibmad_port *srcport) function > > which mirrors madrpc_portid(void) > > Mark all "old" functions with __attribute__ ((deprecated)) > > This generates a lot of warnings right now (even after all patch series > applying it still have deprecated usages in libibmad itself). And this > is not very good. I think our flow should have opposite direction - first > to convert, then mark deprecated functions. > > Now as fast workaround I can mask depreciation by macro: > > #define DEPRECATED /* __attribute__ ((deprecated)) */ > > , and we will uncomment this when everything in tree will be converted. What if we put this #def in the internal *.c files only? Then it will give a warning to users but not on our internal code? I just want to give potential users of the lib time and notice to change their code. > > Also after looking over patch series I see that all "original" function > names become deprecated and replaces by its *_via() brothers. How do > you see the next step? Will we remove old names and have almost all API > calls with useless then _via suffix? Unless you want to break the API I don't see a way around this. "_via" or some other "useless" name change will have to be used? I simply used "_via" because it was already there. I don't know of many people using the lib so if you want we could just change the "original" functions, break the API, and bump the library version. If you don't think there are many users this would be cleaner going forward. But I was trying to move more slowly than that. Ira > > Sasha > > > > > Signed-off-by: Ira Weiny > > --- > > libibmad/include/infiniband/mad.h | 139 ++++++++++++++++++++++--------------- > > libibmad/src/gs.c | 19 +++--- > > libibmad/src/libibmad.map | 1 + > > libibmad/src/resolve.c | 10 ++- > > libibmad/src/rpc.c | 29 ++++---- > > libibmad/src/sa.c | 4 +- > > libibmad/src/smp.c | 4 +- > > 7 files changed, 118 insertions(+), 88 deletions(-) > > > > diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h > > index 1aaaa1b..80e38be 100644 > > --- a/libibmad/include/infiniband/mad.h > > +++ b/libibmad/include/infiniband/mad.h > > @@ -724,100 +724,125 @@ static inline int mad_is_vendor_range2(int mgmt) > > } > > > > /* rpc.c */ > > -MAD_EXPORT int madrpc_portid(void); > > -MAD_EXPORT int madrpc_set_retries(int retries); > > -MAD_EXPORT int madrpc_set_timeout(int timeout); > > -void *madrpc(ib_rpc_t * rpc, ib_portid_t * dport, void *payload, void *rcvdata); > > -void *madrpc_rmpp(ib_rpc_t * rpc, ib_portid_t * dport, ib_rmpp_hdr_t * rmpp, > > - void *data); > > +MAD_EXPORT int madrpc_portid(void) __attribute__ ((deprecated)); > > +void *madrpc(ib_rpc_t * rpc, ib_portid_t * dport, void *payload, void *rcvdata) > > + __attribute__ ((deprecated)); > > +void *madrpc_rmpp(ib_rpc_t * rpc, ib_portid_t * dport, ib_rmpp_hdr_t * rmpp, void *data) > > + __attribute__ ((deprecated)); > > MAD_EXPORT void madrpc_init(char *dev_name, int dev_port, int *mgmt_classes, > > - int num_classes); > > -void madrpc_save_mad(void *madbuf, int len); > > -MAD_EXPORT void madrpc_show_errors(int set); > > + int num_classes) __attribute__ ((deprecated)); > > +void madrpc_save_mad(void *madbuf, int len) __attribute__ ((deprecated)); > > > > -void *mad_rpc_open_port(char *dev_name, int dev_port, int *mgmt_classes, > > +/* New interface */ > > +MAD_EXPORT void madrpc_show_errors(int set); > > +MAD_EXPORT int madrpc_set_retries(int retries); > > +MAD_EXPORT int madrpc_set_timeout(int timeout); > > +MAD_EXPORT struct ibmad_port *mad_rpc_open_port(char *dev_name, int dev_port, int *mgmt_classes, > > int num_classes); > > -void mad_rpc_close_port(void *ibmad_port); > > -void *mad_rpc(const void *ibmad_port, ib_rpc_t * rpc, ib_portid_t * dport, > > - void *payload, void *rcvdata); > > -void *mad_rpc_rmpp(const void *ibmad_port, ib_rpc_t * rpc, ib_portid_t * dport, > > - ib_rmpp_hdr_t * rmpp, void *data); > > +MAD_EXPORT void mad_rpc_close_port(struct ibmad_port *srcport); > > +MAD_EXPORT void *mad_rpc(const struct ibmad_port *srcport, ib_rpc_t * rpc, ib_portid_t * dport, > > + void *payload, void *rcvdata); > > +MAD_EXPORT void *mad_rpc_rmpp(const struct ibmad_port *srcport, ib_rpc_t * rpc, ib_portid_t * dport, > > + ib_rmpp_hdr_t * rmpp, void *data); > > +MAD_EXPORT int mad_rpc_portid(struct ibmad_port *srcport); > > > > /* smp.c */ > > MAD_EXPORT uint8_t *smp_query(void *buf, ib_portid_t * id, unsigned attrid, > > - unsigned mod, unsigned timeout); > > + unsigned mod, unsigned timeout) __attribute__ ((deprecated)); > > MAD_EXPORT uint8_t *smp_set(void *buf, ib_portid_t * id, unsigned attrid, > > - unsigned mod, unsigned timeout); > > + unsigned mod, unsigned timeout) __attribute__ ((deprecated)); > > + > > +/* smp.c new interface */ > > MAD_EXPORT uint8_t *smp_query_via(void *buf, ib_portid_t * id, unsigned attrid, > > - unsigned mod, unsigned timeout, const void *srcport); > > -uint8_t *smp_set_via(void *buf, ib_portid_t * id, unsigned attrid, unsigned mod, > > - unsigned timeout, const void *srcport); > > + unsigned mod, unsigned timeout, const struct ibmad_port *srcport); > > +MAD_EXPORT uint8_t *smp_set_via(void *buf, ib_portid_t * id, unsigned attrid, unsigned mod, > > + unsigned timeout, const struct ibmad_port *srcport); > > > > /* sa.c */ > > uint8_t *sa_call(void *rcvbuf, ib_portid_t * portid, ib_sa_call_t * sa, > > - unsigned timeout); > > -uint8_t *sa_rpc_call(const void *ibmad_port, void *rcvbuf, ib_portid_t * portid, > > + unsigned timeout) __attribute__ ((deprecated)); > > +MAD_EXPORT int ib_path_query(ibmad_gid_t srcgid, ibmad_gid_t destgid, ib_portid_t * sm_id, > > + void *buf) __attribute__ ((deprecated)); > > + > > +/* sa.c new interface */ > > +MAD_EXPORT uint8_t *sa_rpc_call(const struct ibmad_port *srcport, void *rcvbuf, ib_portid_t * portid, > > ib_sa_call_t * sa, unsigned timeout); > > -MAD_EXPORT int ib_path_query(ibmad_gid_t srcgid, ibmad_gid_t destgid, ib_portid_t * sm_id, void *buf); /* returns lid */ > > -int ib_path_query_via(const void *srcport, ibmad_gid_t srcgid, > > +MAD_EXPORT int ib_path_query_via(const struct ibmad_port *srcport, ibmad_gid_t srcgid, > > ibmad_gid_t destgid, ib_portid_t * sm_id, void *buf); > > + /* returns lid */ > > > > /* resolve.c */ > > -MAD_EXPORT int ib_resolve_smlid(ib_portid_t * sm_id, int timeout); > > +MAD_EXPORT int ib_resolve_smlid(ib_portid_t * sm_id, int timeout) > > + __attribute__ ((deprecated)); > > MAD_EXPORT int ib_resolve_guid(ib_portid_t * portid, uint64_t * guid, > > - ib_portid_t * sm_id, int timeout); > > + ib_portid_t * sm_id, int timeout) > > + __attribute__ ((deprecated)); > > MAD_EXPORT int ib_resolve_portid_str(ib_portid_t * portid, char *addr_str, > > - enum MAD_DEST dest, ib_portid_t * sm_id); > > + enum MAD_DEST dest, ib_portid_t * sm_id) > > + __attribute__ ((deprecated)); > > MAD_EXPORT int ib_resolve_self(ib_portid_t * portid, int *portnum, > > - ibmad_gid_t * gid); > > - > > -int ib_resolve_smlid_via(ib_portid_t * sm_id, int timeout, const void *srcport); > > -int ib_resolve_guid_via(ib_portid_t * portid, uint64_t * guid, > > - ib_portid_t * sm_id, int timeout, const void *srcport); > > -int ib_resolve_portid_str_via(ib_portid_t * portid, char *addr_str, > > + ibmad_gid_t * gid) > > + __attribute__ ((deprecated)); > > + > > +/* resolve.c new interface */ > > +MAD_EXPORT int ib_resolve_smlid_via(ib_portid_t * sm_id, int timeout, > > + const struct ibmad_port *srcport); > > +MAD_EXPORT int ib_resolve_guid_via(ib_portid_t * portid, uint64_t * guid, > > + ib_portid_t * sm_id, int timeout, > > + const struct ibmad_port *srcport); > > +MAD_EXPORT int ib_resolve_portid_str_via(ib_portid_t * portid, char *addr_str, > > enum MAD_DEST dest, ib_portid_t * sm_id, > > - const void *srcport); > > -int ib_resolve_self_via(ib_portid_t * portid, int *portnum, ibmad_gid_t * gid, > > - const void *srcport); > > + const struct ibmad_port *srcport); > > +MAD_EXPORT int ib_resolve_self_via(ib_portid_t * portid, int *portnum, ibmad_gid_t * gid, > > + const struct ibmad_port *srcport); > > > > /* gs.c */ > > MAD_EXPORT uint8_t *perf_classportinfo_query(void *rcvbuf, ib_portid_t * dest, > > - int port, unsigned timeout); > > + int port, unsigned timeout) > > + __attribute__ ((deprecated)); > > MAD_EXPORT uint8_t *port_performance_query(void *rcvbuf, ib_portid_t * dest, > > - int port, unsigned timeout); > > + int port, unsigned timeout) > > + __attribute__ ((deprecated)); > > MAD_EXPORT uint8_t *port_performance_reset(void *rcvbuf, ib_portid_t * dest, > > int port, unsigned mask, > > - unsigned timeout); > > + unsigned timeout) > > + __attribute__ ((deprecated)); > > MAD_EXPORT uint8_t *port_performance_ext_query(void *rcvbuf, ib_portid_t * dest, > > - int port, unsigned timeout); > > + int port, unsigned timeout) > > + __attribute__ ((deprecated)); > > MAD_EXPORT uint8_t *port_performance_ext_reset(void *rcvbuf, ib_portid_t * dest, > > int port, unsigned mask, > > - unsigned timeout); > > + unsigned timeout) > > + __attribute__ ((deprecated)); > > MAD_EXPORT uint8_t *port_samples_control_query(void *rcvbuf, ib_portid_t * dest, > > - int port, unsigned timeout); > > + int port, unsigned timeout) > > + __attribute__ ((deprecated)); > > MAD_EXPORT uint8_t *port_samples_result_query(void *rcvbuf, ib_portid_t * dest, > > - int port, unsigned timeout); > > + int port, unsigned timeout) > > + __attribute__ ((deprecated)); > > > > -uint8_t *perf_classportinfo_query_via(void *rcvbuf, ib_portid_t * dest, > > +/* gs.c new interface */ > > +MAD_EXPORT uint8_t *perf_classportinfo_query_via(void *rcvbuf, ib_portid_t * dest, > > int port, unsigned timeout, > > - const void *srcport); > > -uint8_t *port_performance_query_via(void *rcvbuf, ib_portid_t * dest, int port, > > - unsigned timeout, const void *srcport); > > -uint8_t *port_performance_reset_via(void *rcvbuf, ib_portid_t * dest, int port, > > + const struct ibmad_port *srcport); > > +MAD_EXPORT uint8_t *port_performance_query_via(void *rcvbuf, ib_portid_t * dest, int port, > > + unsigned timeout, const struct ibmad_port *srcport); > > +MAD_EXPORT uint8_t *port_performance_reset_via(void *rcvbuf, ib_portid_t * dest, int port, > > unsigned mask, unsigned timeout, > > - const void *srcport); > > -uint8_t *port_performance_ext_query_via(void *rcvbuf, ib_portid_t * dest, > > + const struct ibmad_port *srcport); > > +MAD_EXPORT uint8_t *port_performance_ext_query_via(void *rcvbuf, ib_portid_t * dest, > > int port, unsigned timeout, > > - const void *srcport); > > -uint8_t *port_performance_ext_reset_via(void *rcvbuf, ib_portid_t * dest, > > + const struct ibmad_port *srcport); > > +MAD_EXPORT uint8_t *port_performance_ext_reset_via(void *rcvbuf, ib_portid_t * dest, > > int port, unsigned mask, > > - unsigned timeout, const void *srcport); > > -uint8_t *port_samples_control_query_via(void *rcvbuf, ib_portid_t * dest, > > + unsigned timeout, > > + const struct ibmad_port *srcport); > > +MAD_EXPORT uint8_t *port_samples_control_query_via(void *rcvbuf, ib_portid_t * dest, > > int port, unsigned timeout, > > - const void *srcport); > > -uint8_t *port_samples_result_query_via(void *rcvbuf, ib_portid_t * dest, > > + const struct ibmad_port *srcport); > > +MAD_EXPORT uint8_t *port_samples_result_query_via(void *rcvbuf, ib_portid_t * dest, > > int port, unsigned timeout, > > - const void *srcport); > > + const struct ibmad_port *srcport); > > /* dump.c */ > > MAD_EXPORT ib_mad_dump_fn > > mad_dump_int, mad_dump_uint, mad_dump_hex, mad_dump_rhex, > > diff --git a/libibmad/src/gs.c b/libibmad/src/gs.c > > index d2c4574..e302caf 100644 > > --- a/libibmad/src/gs.c > > +++ b/libibmad/src/gs.c > > @@ -47,7 +47,7 @@ > > > > static uint8_t *pma_query_via(void *rcvbuf, ib_portid_t * dest, int port, > > unsigned timeout, unsigned id, > > - const void *srcport) > > + const struct ibmad_port *srcport) > > { > > ib_rpc_t rpc = { 0 }; > > int lid = dest->lid; > > @@ -89,7 +89,7 @@ uint8_t *pma_query(void *rcvbuf, ib_portid_t * dest, int port, unsigned timeout, > > > > uint8_t *perf_classportinfo_query_via(void *rcvbuf, ib_portid_t * dest, > > int port, unsigned timeout, > > - const void *srcport) > > + const struct ibmad_port *srcport) > > { > > return pma_query_via(rcvbuf, dest, port, timeout, CLASS_PORT_INFO, > > srcport); > > @@ -102,7 +102,7 @@ uint8_t *perf_classportinfo_query(void *rcvbuf, ib_portid_t * dest, int port, > > } > > > > uint8_t *port_performance_query_via(void *rcvbuf, ib_portid_t * dest, int port, > > - unsigned timeout, const void *srcport) > > + unsigned timeout, const struct ibmad_port *srcport) > > { > > return pma_query_via(rcvbuf, dest, port, timeout, > > IB_GSI_PORT_COUNTERS, srcport); > > @@ -116,7 +116,7 @@ uint8_t *port_performance_query(void *rcvbuf, ib_portid_t * dest, int port, > > > > static uint8_t *performance_reset_via(void *rcvbuf, ib_portid_t * dest, > > int port, unsigned mask, unsigned timeout, > > - unsigned id, const void *srcport) > > + unsigned id, const struct ibmad_port *srcport) > > { > > ib_rpc_t rpc = { 0 }; > > int lid = dest->lid; > > @@ -166,7 +166,7 @@ static uint8_t *performance_reset(void *rcvbuf, ib_portid_t * dest, int port, > > > > uint8_t *port_performance_reset_via(void *rcvbuf, ib_portid_t * dest, int port, > > unsigned mask, unsigned timeout, > > - const void *srcport) > > + const struct ibmad_port *srcport) > > { > > return performance_reset_via(rcvbuf, dest, port, mask, timeout, > > IB_GSI_PORT_COUNTERS, srcport); > > @@ -181,7 +181,7 @@ uint8_t *port_performance_reset(void *rcvbuf, ib_portid_t * dest, int port, > > > > uint8_t *port_performance_ext_query_via(void *rcvbuf, ib_portid_t * dest, > > int port, unsigned timeout, > > - const void *srcport) > > + const struct ibmad_port *srcport) > > { > > return pma_query_via(rcvbuf, dest, port, timeout, > > IB_GSI_PORT_COUNTERS_EXT, srcport); > > @@ -195,7 +195,8 @@ uint8_t *port_performance_ext_query(void *rcvbuf, ib_portid_t * dest, int port, > > > > uint8_t *port_performance_ext_reset_via(void *rcvbuf, ib_portid_t * dest, > > int port, unsigned mask, > > - unsigned timeout, const void *srcport) > > + unsigned timeout, > > + const struct ibmad_port *srcport) > > { > > return performance_reset_via(rcvbuf, dest, port, mask, timeout, > > IB_GSI_PORT_COUNTERS_EXT, srcport); > > @@ -210,7 +211,7 @@ uint8_t *port_performance_ext_reset(void *rcvbuf, ib_portid_t * dest, int port, > > > > uint8_t *port_samples_control_query_via(void *rcvbuf, ib_portid_t * dest, > > int port, unsigned timeout, > > - const void *srcport) > > + const struct ibmad_port *srcport) > > { > > return pma_query_via(rcvbuf, dest, port, timeout, > > IB_GSI_PORT_SAMPLES_CONTROL, srcport); > > @@ -225,7 +226,7 @@ uint8_t *port_samples_control_query(void *rcvbuf, ib_portid_t * dest, int port, > > > > uint8_t *port_samples_result_query_via(void *rcvbuf, ib_portid_t * dest, > > int port, unsigned timeout, > > - const void *srcport) > > + const struct ibmad_port *srcport) > > { > > return pma_query_via(rcvbuf, dest, port, timeout, > > IB_GSI_PORT_SAMPLES_RESULT, srcport); > > diff --git a/libibmad/src/libibmad.map b/libibmad/src/libibmad.map > > index f944d86..94d7762 100644 > > --- a/libibmad/src/libibmad.map > > +++ b/libibmad/src/libibmad.map > > @@ -69,6 +69,7 @@ IBMAD_1.3 { > > mad_rpc_close_port; > > mad_rpc; > > mad_rpc_rmpp; > > + mad_rpc_portid; > > madrpc; > > madrpc_def_timeout; > > madrpc_init; > > diff --git a/libibmad/src/resolve.c b/libibmad/src/resolve.c > > index 553949d..3291f43 100644 > > --- a/libibmad/src/resolve.c > > +++ b/libibmad/src/resolve.c > > @@ -45,7 +45,8 @@ > > #undef DEBUG > > #define DEBUG if (ibdebug) IBWARN > > > > -int ib_resolve_smlid_via(ib_portid_t * sm_id, int timeout, const void *srcport) > > +int ib_resolve_smlid_via(ib_portid_t * sm_id, int timeout, > > + const struct ibmad_port *srcport) > > { > > ib_portid_t self = { 0 }; > > uint8_t portinfo[64]; > > @@ -67,7 +68,8 @@ int ib_resolve_smlid(ib_portid_t * sm_id, int timeout) > > } > > > > int ib_resolve_guid_via(ib_portid_t * portid, uint64_t * guid, > > - ib_portid_t * sm_id, int timeout, const void *srcport) > > + ib_portid_t * sm_id, int timeout, > > + const struct ibmad_port *srcport) > > { > > ib_portid_t sm_portid; > > char buf[IB_SA_DATA_SIZE] = { 0 }; > > @@ -93,7 +95,7 @@ int ib_resolve_guid_via(ib_portid_t * portid, uint64_t * guid, > > > > int ib_resolve_portid_str_via(ib_portid_t * portid, char *addr_str, > > enum MAD_DEST dest_type, ib_portid_t * sm_id, > > - const void *srcport) > > + const struct ibmad_port *srcport) > > { > > uint64_t guid; > > int lid; > > @@ -150,7 +152,7 @@ int ib_resolve_portid_str(ib_portid_t * portid, char *addr_str, > > } > > > > int ib_resolve_self_via(ib_portid_t * portid, int *portnum, ibmad_gid_t * gid, > > - const void *srcport) > > + const struct ibmad_port *srcport) > > { > > ib_portid_t self = { 0 }; > > uint8_t portinfo[64]; > > diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c > > index e811526..d47873b 100644 > > --- a/libibmad/src/rpc.c > > +++ b/libibmad/src/rpc.c > > @@ -100,6 +100,11 @@ int madrpc_portid(void) > > return mad_portid; > > } > > > > +int mad_rpc_portid(struct ibmad_port *srcport) > > +{ > > + return (srcport->port_id); > > +} > > + > > static int > > _do_madrpc(int port_id, void *sndbuf, void *rcvbuf, int agentid, int len, > > int timeout) > > @@ -164,10 +169,9 @@ _do_madrpc(int port_id, void *sndbuf, void *rcvbuf, int agentid, int len, > > return -1; > > } > > > > -void *mad_rpc(const void *port_id, ib_rpc_t * rpc, ib_portid_t * dport, > > +void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport, > > void *payload, void *rcvdata) > > { > > - const struct ibmad_port *p = port_id; > > int status, len; > > uint8_t sndbuf[1024], rcvbuf[1024], *mad; > > > > @@ -177,8 +181,8 @@ void *mad_rpc(const void *port_id, ib_rpc_t * rpc, ib_portid_t * dport, > > if ((len = mad_build_pkt(sndbuf, rpc, dport, 0, payload)) < 0) > > return 0; > > > > - if ((len = _do_madrpc(p->port_id, sndbuf, rcvbuf, > > - p->class_agents[rpc->mgtclass], > > + if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, > > + port->class_agents[rpc->mgtclass], > > len, rpc->timeout)) < 0) { > > IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); > > return 0; > > @@ -203,10 +207,9 @@ void *mad_rpc(const void *port_id, ib_rpc_t * rpc, ib_portid_t * dport, > > return rcvdata; > > } > > > > -void *mad_rpc_rmpp(const void *port_id, ib_rpc_t * rpc, ib_portid_t * dport, > > +void *mad_rpc_rmpp(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport, > > ib_rmpp_hdr_t * rmpp, void *data) > > { > > - const struct ibmad_port *p = port_id; > > int status, len; > > uint8_t sndbuf[1024], rcvbuf[1024], *mad; > > > > @@ -217,8 +220,8 @@ void *mad_rpc_rmpp(const void *port_id, ib_rpc_t * rpc, ib_portid_t * dport, > > if ((len = mad_build_pkt(sndbuf, rpc, dport, rmpp, data)) < 0) > > return 0; > > > > - if ((len = _do_madrpc(p->port_id, sndbuf, rcvbuf, > > - p->class_agents[rpc->mgtclass], > > + if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, > > + port->class_agents[rpc->mgtclass], > > len, rpc->timeout)) < 0) { > > IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); > > return 0; > > @@ -303,7 +306,7 @@ madrpc_init(char *dev_name, int dev_port, int *mgmt_classes, int num_classes) > > } > > } > > > > -void *mad_rpc_open_port(char *dev_name, int dev_port, > > +struct ibmad_port *mad_rpc_open_port(char *dev_name, int dev_port, > > int *mgmt_classes, int num_classes) > > { > > struct ibmad_port *p; > > @@ -360,12 +363,10 @@ void *mad_rpc_open_port(char *dev_name, int dev_port, > > return p; > > } > > > > -void mad_rpc_close_port(void *port_id) > > +void mad_rpc_close_port(struct ibmad_port *port) > > { > > - struct ibmad_port *p = port_id; > > - > > - umad_close_port(p->port_id); > > - free(p); > > + umad_close_port(port->port_id); > > + free(port); > > } > > > > uint8_t *sa_call(void *rcvbuf, ib_portid_t * portid, ib_sa_call_t * sa, > > diff --git a/libibmad/src/sa.c b/libibmad/src/sa.c > > index 7403d4f..ddeb152 100644 > > --- a/libibmad/src/sa.c > > +++ b/libibmad/src/sa.c > > @@ -44,7 +44,7 @@ > > #undef DEBUG > > #define DEBUG if (ibdebug) IBWARN > > > > -uint8_t *sa_rpc_call(const void *ibmad_port, void *rcvbuf, ib_portid_t * portid, > > +uint8_t *sa_rpc_call(const struct ibmad_port *ibmad_port, void *rcvbuf, ib_portid_t * portid, > > ib_sa_call_t * sa, unsigned timeout) > > { > > ib_rpc_t rpc = { 0 }; > > @@ -106,7 +106,7 @@ uint8_t *sa_rpc_call(const void *ibmad_port, void *rcvbuf, ib_portid_t * portid, > > IB_PR_COMPMASK_SGID |\ > > IB_PR_COMPMASK_NUMBPATH) > > > > -int ib_path_query_via(const void *srcport, ibmad_gid_t srcgid, > > +int ib_path_query_via(const struct ibmad_port *srcport, ibmad_gid_t srcgid, > > ibmad_gid_t destgid, ib_portid_t * sm_id, void *buf) > > { > > int npath; > > diff --git a/libibmad/src/smp.c b/libibmad/src/smp.c > > index fad263c..e5489b3 100644 > > --- a/libibmad/src/smp.c > > +++ b/libibmad/src/smp.c > > @@ -45,7 +45,7 @@ > > #define DEBUG if (ibdebug) IBWARN > > > > uint8_t *smp_set_via(void *data, ib_portid_t * portid, unsigned attrid, > > - unsigned mod, unsigned timeout, const void *srcport) > > + unsigned mod, unsigned timeout, const struct ibmad_port *srcport) > > { > > ib_rpc_t rpc = { 0 }; > > > > @@ -81,7 +81,7 @@ uint8_t *smp_set(void *data, ib_portid_t * portid, unsigned attrid, > > } > > > > uint8_t *smp_query_via(void *rcvbuf, ib_portid_t * portid, unsigned attrid, > > - unsigned mod, unsigned timeout, const void *srcport) > > + unsigned mod, unsigned timeout, const struct ibmad_port *srcport) > > { > > ib_rpc_t rpc = { 0 }; > > > > -- > > 1.5.4.5 > > -- Ira Weiny Math Programer/Computer Scientist Larence Livermore National Lab weiny2 at llnl.gov From weiny2 at llnl.gov Mon Mar 2 09:56:02 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Mon, 2 Mar 2009 09:56:02 -0800 Subject: [ofa-general] [PATCH 11/10] libibmad:infiniband-diags: deprecate madrpc_set_[retries|timeout] WAS: [PATCH 1/10] libibmad: Clean up "new" interface In-Reply-To: <20090301072622.GG3936@sashak.voltaire.com> References: <20090219190525.322681b8.weiny2@llnl.gov> <20090220143402.c3b23b0a.weiny2@llnl.gov> <20090301072622.GG3936@sashak.voltaire.com> Message-ID: <20090302095602.b1062edd.weiny2@llnl.gov> On Sun, 1 Mar 2009 09:26:31 +0200 Sasha Khapyorsky wrote: > On 14:34 Fri 20 Feb , Ira Weiny wrote: [snip] > > diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c > > index 210f0c2..229020d 100644 > > --- a/libibmad/src/rpc.c > > +++ b/libibmad/src/rpc.c > > @@ -49,7 +49,7 @@ int ibdebug; > > > > static int mad_portid = -1; > > static int iberrs; > > - > > + int timeout; > > Typo? yes sorry. Ira [snip] From weiny2 at llnl.gov Mon Mar 2 11:11:01 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Mon, 2 Mar 2009 11:11:01 -0800 Subject: [ofa-general] [PATCH 11/10] libibmad:infiniband-diags: deprecate madrpc_set_[retries|timeout] WAS: [PATCH 1/10] libibmad: Clean up "new" interface In-Reply-To: References: <20090219190525.322681b8.weiny2@llnl.gov> <20090220143402.c3b23b0a.weiny2@llnl.gov> <20090301072622.GG3936@sashak.voltaire.com> Message-ID: <20090302111101.bc822db8.weiny2@llnl.gov> On Sun, 1 Mar 2009 20:23:59 -0500 Hal Rosenstock wrote: > On Sun, Mar 1, 2009 at 2:26 AM, Sasha Khapyorsky wrote: > > [snip...] > > > Now you need to duplicate this single call over all tools. For me it > > looks like an overkill. > > See below. > > > Probably it would be simpler to just read global > > ibd_timeout variable in rpc.c? > > [snip...] > > > Same with retries - it is hard for me to believe that any multithreaded > > application will try to setup different retry values per port, for > > different threads, "on the fly".... > > For timeouts and retries, per port makes sense to me as ports may be > connected to different fabrics with different timeouts. Also, it may > depend on the operation (e.g. restricted to a single subnet or across > multiple subnets). I see this as well. Right now libibmad is designed around the "run and exit" diag model but we are moving it toward a library which can be used in more complex applications. We should push to do this once and as correct as possible. > > > (rpc.c with all its limited functionality will not be sufficient for such flexibility > > level anyway :)). > > This would be an implementation issue to fix if the more flexible > timeout/retries is to be implemented. Agreed. I have found further places where the timeout needs to be cleaned up. Just search for timeout in mad.h... In addition to the *_set_timeout calls; there are many functions which specify it directly; either through a struct (ib_rpc_t, ib_vendor_call_t) or via parameter (mad_receive*, smp_[query|set], sa_call, sa_rcp_call, ib_resolve*, pma_query_via, performance_reset_via). Retries are a bit more straight forward. I will think on this some more, Ira -- Ira Weiny Math Programer/Computer Scientist Larence Livermore National Lab weiny2 at llnl.gov From hnrose at comcast.net Mon Mar 2 13:01:25 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Mon, 2 Mar 2009 16:01:25 -0500 Subject: [ofa-general] ***SPAM*** [PATCH][TRIVIAL] opensm/osm_port.h: Fix a commentary typo Message-ID: <20090302210125.GB7285@comcast.net> Also, cosmetic formatting change Signed-off-by: Hal Rosenstock --- diff --git a/opensm/include/opensm/osm_port.h b/opensm/include/opensm/osm_port.h index 3269519..7079e74 100644 --- a/opensm/include/opensm/osm_port.h +++ b/opensm/include/opensm/osm_port.h @@ -1144,7 +1144,7 @@ static inline osm_dr_path_t *osm_physp_get_dr_path_ptr(IN const osm_physp_t * * osm_port_t * * DESCRIPTION -* This object represents a logical port on a switch, router or end-point. +* This object represents a logical port on a switch, router, or CA. * * The osm_port_t object should be treated as opaque and should * be manipulated only through the provided functions. @@ -1169,7 +1169,8 @@ typedef struct osm_port { * Linkage structure for cl_qmap. MUST BE FIRST MEMBER! * * list_item -* Linkage structure for cl_qlist. Used by ucast mgr during LFT calculation. +* Linkage structure for cl_qlist. Used by ucast mgr during +* LFT calculation. * * p_node * Points to the Node object that owns this port. From hnrose at comcast.net Mon Mar 2 13:00:29 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Mon, 2 Mar 2009 16:00:29 -0500 Subject: [ofa-general] [PATCH] libibumad/umad.c: Cosmetic changes Message-ID: <20090302210029.GA7285@comcast.net> In get_port, change variable name for better readability In resolve_ca_name, use define rather than constant in umad_get_cas_names call Signed-off-by: Hal Rosenstock --- diff --git a/libibumad/src/umad.c b/libibumad/src/umad.c index 017f6be..72ef506 100644 --- a/libibumad/src/umad.c +++ b/libibumad/src/umad.c @@ -136,7 +136,7 @@ get_port(char *ca_name, char *dir, int portnum, umad_port_t *port) char port_dir[256]; uint8_t gid[16]; struct dirent **namelist = NULL; - int i, len, ret = 0; + int i, len, num_pkeys = 0; strncpy(port->ca_name, ca_name, sizeof port->ca_name - 1); port->portnum = portnum; @@ -172,25 +172,25 @@ get_port(char *ca_name, char *dir, int portnum, umad_port_t *port) memcpy(&port->port_guid, gid + 8, sizeof port->port_guid); snprintf(port_dir + len, sizeof(port_dir) - len, "/pkeys"); - ret = scandir(port_dir, &namelist, check_for_digit_name, NULL); - if (ret <= 0) { + num_pkeys = scandir(port_dir, &namelist, check_for_digit_name, NULL); + if (num_pkeys <= 0) { IBWARN("no pkeys found for %s:%u (at dir %s)...", port->ca_name, port->portnum, port_dir); goto clean; } - port->pkeys = calloc(ret, sizeof(port->pkeys[0])); + port->pkeys = calloc(num_pkeys, sizeof(port->pkeys[0])); if (!port->pkeys) { IBWARN("get_port: calloc failed: %s", strerror(errno)); goto clean; } - for (i = 0; i < ret ; i++) { + for (i = 0; i < num_pkeys; i++) { unsigned idx, val; idx = strtoul(namelist[i]->d_name, NULL, 0); sys_read_uint(port_dir, namelist[i]->d_name, &val); port->pkeys[idx] = val; free(namelist[i]); } - port->pkeys_size = ret; + port->pkeys_size = num_pkeys; free(namelist); namelist = NULL; port_dir[len] = '\0'; @@ -201,7 +201,7 @@ get_port(char *ca_name, char *dir, int portnum, umad_port_t *port) clean: if (namelist) { - for (i = 0; i < ret ; i++) + for (i = 0; i < num_pkeys; i++) free(namelist[i]); free(namelist); } @@ -311,7 +311,7 @@ resolve_ca_name(char *ca_name, int *best_port) } /* Get the list of CA names */ - if ((n = umad_get_cas_names((void *)names, 20)) < 0) + if ((n = umad_get_cas_names((void *)names, UMAD_MAX_DEVICES)) < 0) return 0; /* Find the first existing CA with an active port */ From hnrose at comcast.net Mon Mar 2 13:02:29 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Mon, 2 Mar 2009 16:02:29 -0500 Subject: [ofa-general] ***SPAM*** [PATCH][TRIVIAL] opensm/osm_state_mgr.c: Cosmetic commentary change Message-ID: <20090302210229.GC7285@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c index 0d7cf15..fe516a6 100644 --- a/opensm/opensm/osm_state_mgr.c +++ b/opensm/opensm/osm_state_mgr.c @@ -513,8 +513,8 @@ static void query_sm_info(cl_map_item_t *item, void *cxt) } /********************************************************************** - During a light sweep check each node to see if the node descriptor is valid - if not issue a ND query. + During a light sweep, check each node to see if the node description + is valid and if not issue a ND query. **********************************************************************/ static void __osm_state_mgr_get_node_desc(IN cl_map_item_t * const p_object, IN void *context) From weiny2 at llnl.gov Mon Mar 2 16:06:59 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Mon, 2 Mar 2009 16:06:59 -0800 Subject: [ofa-general] Re: [PATCH 2/10] infiniband-diags: Convert ibaddr to "new" ibmad interface In-Reply-To: <20090301080804.GH3936@sashak.voltaire.com> References: <20090219190528.11c080f8.weiny2@llnl.gov> <20090301080804.GH3936@sashak.voltaire.com> Message-ID: <20090302160659.0807fb24.weiny2@llnl.gov> On Sun, 1 Mar 2009 10:08:04 +0200 Sasha Khapyorsky wrote: > On 19:05 Thu 19 Feb , Ira Weiny wrote: [snip] > > > > - madrpc_init(ibd_ca, ibd_ca_port, mgmt_classes, 3); > > + srcport = mad_rpc_open_port(ibd_ca, ibd_ca_port, mgmt_classes, 3); > > + if (!srcport) > > + IBERROR("Failed to open '%s' port '%d'", ibd_ca, ibd_ca_port); > > Here and in almost all other infiniband-diags patches: > > madrpc_init() in case of failures calls IBPANIC() which prints error > message and calls exit(). Unlike this mad_rpc_open_port() sets errno and > returns NULL, IBERROR() just prints error message - the program will > continue execution... Good catch I will update with a proper exit call. [snip] > > diff --git a/infiniband-diags/src/ibdiag_common.c b/infiniband-diags/src/ibdiag_common.c > > index 5f2472d..609df69 100644 > > --- a/infiniband-diags/src/ibdiag_common.c > > +++ b/infiniband-diags/src/ibdiag_common.c > > @@ -179,7 +179,8 @@ static int process_opt(int ch, char *optarg) > > ibd_timeout = val; > > break; > > case 's': > > - if (ib_resolve_portid_str(&sm_portid, optarg, IB_DEST_LID, 0) < 0) > > + if (ib_resolve_portid_str_via(&sm_portid, optarg, IB_DEST_LID, > > + 0, NULL) < 0) > > This may be not correct (not introduced by those patches) - then host is > connected to multiple subnets we need to resolve SM LID on requested > one, eg. using srcport... > Would you accept adding srcport to ibdiag_process_opts? Or would you prefer something more generic? Ira From ralph.campbell at qlogic.com Mon Mar 2 16:57:24 2009 From: ralph.campbell at qlogic.com (Ralph Campbell) Date: Mon, 02 Mar 2009 16:57:24 -0800 Subject: [ofa-general] [PATCH] IB/core: ib_post_send_mad() returns zero but doesn't generate send completion In-Reply-To: References: <1235771157.3948.233.camel@chromite.mv.qlogic.com> Message-ID: <1236041844.3948.293.camel@chromite.mv.qlogic.com> On Sun, 2009-03-01 at 17:23 -0800, Hal Rosenstock wrote: > On Fri, Feb 27, 2009 at 4:45 PM, Ralph Campbell > wrote: > > Good catches. > > > If ib_post_send_mad() returns zero, it guarantees that there will be > > a callback to the send_buf->mad_agent->send_handler() so that the > > sender can call ib_free_send_mad(). Otherwise, the ib_mad_send_buf > > will be leaked and the mad_agent reference count will never go to zero > > and the IB device module cannot be unloaded. > > The above can happen without this patch if process_mad() returns > > (IB_MAD_RESULT_SUCCESS | IB_MAD_RESULT_CONSUMED). > > This looks right although I'm not sure why this wasn't seen a long time ago. mthca returns IB_MAD_RESULT_CONSUMED for TRAP repress. I verified that using libibumad to send a direct routed TRAP repress with hop=1/cnt=0 does hang on mthca. I think opensm only sends LID routed TRAP repress so that would explain why this wasn't seen earlier. For ipath, it was fairly recently that we made SubnSet(Portinfo) to disabled to not send a reply and we have a program that uses libibumad to disable ports. Without the patch, the program hangs. > > If process_mad() returns IB_MAD_RESULT_SUCCESS and there is no agent > > registered to receive the mad being sent, handle_outgoing_dr_smp() > > returns zero which causes a MAD packet which is at the end of the > > directed route to be incorrectly sent on the wire but doesn't cause > > a hang since the HCA generates a send completion. > > This also looks right. > > Does the packet really get out on the IB wire ? Was that with a > Mellanox HCA ? I don't recall ever seeing these extra packets but > maybe they end up swallowed by certain HCAs. Yes, a packet is sent with either mthca or ipath. > Unfortunately, I have no current way of regressing these changes. I'm > presuming you've done this with Mellanox HCAs and with OpenSM running. opensm runs fine with the patch applied on either mthca or ipath. > One other question is whether this should be separated into two increments. I think the changes are small enough and related enough to justify a single patch but I will split it into two if someone feels I should. > -- Hal > > > Signed-off-by: Ralph Campbell > > > > diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c > > index dbcd285..62a99dc 100644 > > --- a/drivers/infiniband/core/mad.c > > +++ b/drivers/infiniband/core/mad.c > > @@ -742,9 +742,7 @@ static int handle_outgoing_dr_smp(struct ib_mad_agent_private *mad_agent_priv, > > break; > > case IB_MAD_RESULT_SUCCESS | IB_MAD_RESULT_CONSUMED: > > kmem_cache_free(ib_mad_cache, mad_priv); > > - kfree(local); > > - ret = 1; > > - goto out; > > + break; > > case IB_MAD_RESULT_SUCCESS: > > /* Treat like an incoming receive MAD */ > > port_priv = ib_get_mad_port(mad_agent_priv->agent.device, > > @@ -755,10 +753,12 @@ static int handle_outgoing_dr_smp(struct ib_mad_agent_private *mad_agent_priv, > > &mad_priv->mad.mad); > > } > > if (!port_priv || !recv_mad_agent) { > > + /* > > + * No receiving agent so drop packet and > > + * generate send completion. > > + */ > > kmem_cache_free(ib_mad_cache, mad_priv); > > - kfree(local); > > - ret = 0; > > - goto out; > > + break; > > } > > local->mad_priv = mad_priv; > > local->recv_mad_agent = recv_mad_agent; > > > > > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > From jackm at dev.mellanox.co.il Mon Mar 2 23:20:46 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Tue, 3 Mar 2009 09:20:46 +0200 Subject: [ofa-general] [PATCH] IB/sa_query: fix update_sm_ah() race condition. Message-ID: <200903030920.46659.jackm@dev.mellanox.co.il> Our testing uncovered a race condition in ib_sa_event(): spin_lock_irqsave(&port->ah_lock, flags); if (port->sm_ah) kref_put(&port->sm_ah->ref, free_sm_ah); port->sm_ah = NULL; spin_unlock_irqrestore(&port->ah_lock, flags); schedule_work(&sa_dev->port[event->element.port_num - sa_dev->start_port].update_task); If two events occur back-to-back (e.g., client-reregister and lid change), both may pass the spinlock-protected code above before the first schedule_work updates the port->sm_ah handle. The second schedule_work operation will then find a non-null port->ah_lock, and will simply overwrite it in update_sm_ah -- resulting in an ah leak. Signed-off-by: Jack Morgenstein diff --git a/drivers/infiniband/core/sa_query.c b/drivers/infiniband/core/sa_query.c index 7863a50..1865049 100644 --- a/drivers/infiniband/core/sa_query.c +++ b/drivers/infiniband/core/sa_query.c @@ -395,6 +395,8 @@ static void update_sm_ah(struct work_struct *work) } spin_lock_irq(&port->ah_lock); + if (port->sm_ah) + kref_put(&port->sm_ah->ref, free_sm_ah); port->sm_ah = new_ah; spin_unlock_irq(&port->ah_lock); From jackm at dev.mellanox.co.il Tue Mar 3 02:14:45 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Tue, 3 Mar 2009 12:14:45 +0200 Subject: [ofa-general] Re: Problem in IB network without Switch In-Reply-To: References: <829ded920902260031r6f8b973t9f2e536864e25c85@mail.gmail.com> <200903021646.47008.jackm@dev.mellanox.co.il> Message-ID: <200903031214.46063.jackm@dev.mellanox.co.il> You need to get in touch with Mellanox Support (FAE) at this point. - Jack On Monday 02 March 2009 17:07, lakshmana swamy wrote: > > Hi Jack, > > > Yes, I connected the cable between two ports of same HCA. Without running opensmd. > > Now the State is " Initializing" > > I observed same state in other node also. > > If cable is bad, it should not show this state right... > > Thanks > laxman > > > > From: jackm at dev.mellanox.co.il > > To: klakshman03 at hotmail.com > > Subject: Re: Problem in IB network without Switch > > Date: Mon, 2 Mar 2009 16:46:46 +0200 > > CC: keshetti.mahesh at gmail.com; general at lists.openfabrics.org > > > > On Monday 02 March 2009 16:38, lakshmana swamy wrote: > > > > > > > > HI Jack > > > > > > I have updated the firmware of HCA in both the machines, but the status remains same. > > > Please have a look at the following outputs. > > > > > > What may be the problem ? > > > > > Your physical connection is bad. Check your cables. > > > > Try connecting Port 1 to Port 2 of the same HCA directly, and see if you still get state: Down. > > (you should see state: init before running opensm). > > > > - Jack > > > > > [root at locluster ~]# ibstat > > > CA 'mthca0' > > > CA type: MT23108 > > > Number of ports: 2 > > > Firmware version: 3.5.917 > > > Hardware version: a1 > > > Node GUID: 0x0002c90108cd13c0 > > > System image GUID: 0x0002c90108cd13c0 > > > Port 1: > > > State: Down > > > Physical state: Polling > > > Rate: 2 > > > Base lid: 0 > > > LMC: 0 > > > SM lid: 0 > > > Capability mask: 0x02510a6a > > > Port GUID: 0x0002c90108cd13c1 > > > Port 2: > > > State: Down > > > Physical state: Polling > > > Rate: 2 > > > Base lid: 0 > > > LMC: 0 > > > SM lid: 0 > > > Capability mask: 0x02510a68 > > > Port GUID: 0x0002c90108cd13c2 > > > [root at locluster ~]# > > > > > > [root at locluster1 ~]# ibstat > > > CA 'mthca0' > > > CA type: MT23108 > > > Number of ports: 2 > > > Firmware version: 3.5.917 > > > Hardware version: a1 > > > Node GUID: 0x0002c9020000114c > > > System image GUID: 0x0002c9020000114f > > > Port 1: > > > State: Down > > > Physical state: Polling > > > Rate: 2 > > > Base lid: 0 > > > LMC: 0 > > > SM lid: 0 > > > Capability mask: 0x02510a68 > > > Port GUID: 0x0002c9020000114d > > > Port 2: > > > State: Down > > > Physical state: Polling > > > Rate: 2 > > > Base lid: 0 > > > LMC: 0 > > > SM lid: 0 > > > Capability mask: 0x02510a68 > > > Port GUID: 0x0002c9020000114e > > > [root at locluster1 ~]# > > > > > > > > > > > > ThankQ > > > > > > laxman > > > > > > > > > From: klakshman03 at hotmail.com > > > To: jackm at dev.mellanox.co.il > > > CC: keshetti.mahesh at gmail.com; general at lists.openfabrics.org > > > Subject: RE: Problem in IB network without Switch > > > Date: Fri, 27 Feb 2009 09:56:38 +0530 > > > > > > > > > > > > > > > > > > > > > > > > > > > ThanQ Jack > > > > > > I will Update the firmware and let you know the status > > > > > > laxman > > > > > > > From: jackm at dev.mellanox.co.il > > > > To: klakshman03 at hotmail.com > > > > Subject: Re: Problem in IB network without Switch > > > > Date: Thu, 26 Feb 2009 18:49:40 +0200 > > > > CC: keshetti.mahesh at gmail.com; general at lists.openfabrics.org > > > > > > > > You are running VERY old firmware (from 2004), and moreover, on one host > > > > you have 3.0.0, and on the other 3.1.0. > > > > > > > > You need to upgrade your firmware. > > > > Contact your Mellanox FAE (support engineer) for instructions. > > > > > > > > - Jack > > > > > > > > > Hi Jack, > > > > > > > > > > Please find the output of ibstat on both the nodes, . > > > > > > > > > > [root at mattool ~]# /opt/ofed/extras/hca_self_test.ofed > > > > > HCA Firmware Check ..................... FAIL > > > > > REASON: mismatch HCA #0 firmware detected (found v, need v3.5.917) > > > > > Host Driver Initialization ............. PASS > > > > > > > > > > [root at mattool ~]# > > > > > > > > > > ************ IBSTAT output ****************** > > > > > > > > > > > > > > > [root at mattool ~]# ibstat > > > > > CA 'mthca0' > > > > > CA type: MT23108 > > > > > Number of ports: 2 > > > > > Firmware version: 3.1.0 > > > > > > > > > [root at compute-0-0 ~]# ibstat > > > > > CA 'mthca0' > > > > > CA type: MT23108 > > > > > Number of ports: 2 > > > > > Firmware version: 3.0.0 > > > > > > Get a view of the world through MSN Video. Some things just cannot be left unseen. Try it! > > > _________________________________________________________________ > > > Express your true emotions! Send a virtual kiss! Do all these cool stuff & more with Windows Live Messenger. Download here > > > http://www.windowslive.com/Desktop/Messenger > > _________________________________________________________________ > Express your true emotions! Send a virtual kiss! Do all these cool stuff & more with Windows Live Messenger. Download here > http://www.windowslive.com/Desktop/Messenger From vlad at lists.openfabrics.org Tue Mar 3 03:19:41 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Tue, 3 Mar 2009 03:19:41 -0800 (PST) Subject: [ofa-general] ofa_1_4_kernel 20090303-0200 daily build status Message-ID: <20090303111941.C4C90E60FC8@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From yevgenyp at mellanox.co.il Tue Mar 3 04:42:54 2009 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Tue, 03 Mar 2009 14:42:54 +0200 Subject: [ofa-general][PATCH 1/2] mlx4_core: Link sensing support Message-ID: <49AD25CE.8010208@mellanox.co.il> At any time when port link is down (except to driver restart), and port is configured to auto sensing mode, we try to sense port link type in order to determine how to initialize the port. If port type needs to be changed, all mlx4 interfaces are unregistered and then registered again with the new port types. Sensing is done with intervals of 3 seconds. Signed-off-by: Yevgeny Petrilin --- drivers/net/mlx4/Makefile | 2 +- drivers/net/mlx4/eq.c | 16 +++- drivers/net/mlx4/intf.c | 4 + drivers/net/mlx4/main.c | 94 +++++++++++++++++------ drivers/net/mlx4/mlx4.h | 26 +++++++ drivers/net/mlx4/sense.c | 174 +++++++++++++++++++++++++++++++++++++++++++ include/linux/mlx4/cmd.h | 1 + include/linux/mlx4/device.h | 6 +- 8 files changed, 291 insertions(+), 32 deletions(-) create mode 100644 drivers/net/mlx4/sense.c diff --git a/drivers/net/mlx4/Makefile b/drivers/net/mlx4/Makefile index a7a97bf..5a07b13 100644 --- a/drivers/net/mlx4/Makefile +++ b/drivers/net/mlx4/Makefile @@ -1,7 +1,7 @@ obj-$(CONFIG_MLX4_CORE) += mlx4_core.o mlx4_core-y := alloc.o catas.o cmd.o cq.o eq.o fw.o icm.o intf.o main.o mcg.o \ - mr.o pd.o port.o profile.o qp.o reset.o srq.o + mr.o pd.o port.o profile.o qp.o reset.o srq.o sense.o obj-$(CONFIG_MLX4_EN) += mlx4_en.o diff --git a/drivers/net/mlx4/eq.c b/drivers/net/mlx4/eq.c index 2c19bff..8830dcb 100644 --- a/drivers/net/mlx4/eq.c +++ b/drivers/net/mlx4/eq.c @@ -163,6 +163,7 @@ static int mlx4_eq_int(struct mlx4_dev *dev, struct mlx4_eq *eq) int cqn; int eqes_found = 0; int set_ci = 0; + int port; while ((eqe = next_eqe_sw(eq))) { /* @@ -203,11 +204,16 @@ static int mlx4_eq_int(struct mlx4_dev *dev, struct mlx4_eq *eq) break; case MLX4_EVENT_TYPE_PORT_CHANGE: - mlx4_dispatch_event(dev, - eqe->subtype == MLX4_PORT_CHANGE_SUBTYPE_ACTIVE ? - MLX4_DEV_EVENT_PORT_UP : - MLX4_DEV_EVENT_PORT_DOWN, - be32_to_cpu(eqe->event.port_change.port) >> 28); + port = be32_to_cpu(eqe->event.port_change.port) >> 28; + if (eqe->subtype == MLX4_PORT_CHANGE_SUBTYPE_DOWN) { + mlx4_dispatch_event(dev, MLX4_DEV_EVENT_PORT_DOWN, + port); + mlx4_priv(dev)->sense.do_sense_port[port] = 1; + } else { + mlx4_dispatch_event(dev, MLX4_DEV_EVENT_PORT_UP, + port); + mlx4_priv(dev)->sense.do_sense_port[port] = 0; + } break; case MLX4_EVENT_TYPE_CQ_ERROR: diff --git a/drivers/net/mlx4/intf.c b/drivers/net/mlx4/intf.c index 0e7eb10..30ef000 100644 --- a/drivers/net/mlx4/intf.c +++ b/drivers/net/mlx4/intf.c @@ -141,6 +141,8 @@ int mlx4_register_device(struct mlx4_dev *dev) mutex_unlock(&intf_mutex); mlx4_start_catas_poll(dev); + mlx4_start_sense(dev); + return 0; } @@ -149,6 +151,8 @@ void mlx4_unregister_device(struct mlx4_dev *dev) struct mlx4_priv *priv = mlx4_priv(dev); struct mlx4_interface *intf; + mlx4_stop_sense(dev); + mlx4_stop_catas_poll(dev); mutex_lock(&intf_mutex); diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c index 8480f03..a50ae7a 100644 --- a/drivers/net/mlx4/main.c +++ b/drivers/net/mlx4/main.c @@ -98,24 +98,23 @@ module_param_named(use_prio, use_prio, bool, 0444); MODULE_PARM_DESC(use_prio, "Enable steering by VLAN priority on ETH ports " "(0/1, default 0)"); -static int mlx4_check_port_params(struct mlx4_dev *dev, +int mlx4_check_port_params(struct mlx4_dev *dev, enum mlx4_port_type *port_type) { int i; for (i = 0; i < dev->caps.num_ports - 1; i++) { - if (port_type[i] != port_type[i+1] && - !(dev->caps.flags & MLX4_DEV_CAP_FLAG_DPDP)) { - mlx4_err(dev, "Only same port types supported " - "on this HCA, aborting.\n"); - return -EINVAL; + if (port_type[i] != port_type[i+1]) { + if (!(dev->caps.flags & MLX4_DEV_CAP_FLAG_DPDP)) { + mlx4_err(dev, "Only same port types supported " + "on this HCA, aborting.\n"); + return -EINVAL; + } + if ((port_type[i] == MLX4_PORT_TYPE_ETH) || + (port_type[i+1] == MLX4_PORT_TYPE_IB)) + return -EINVAL; } } - if ((port_type[0] == MLX4_PORT_TYPE_ETH) && - (port_type[1] == MLX4_PORT_TYPE_IB)) { - mlx4_err(dev, "eth-ib configuration is not supported.\n"); - return -EINVAL; - } for (i = 0; i < dev->caps.num_ports; i++) { if (!(port_type[i] & dev->caps.supported_type[i+1])) { @@ -225,6 +224,9 @@ static int mlx4_dev_cap(struct mlx4_dev *dev, struct mlx4_dev_cap *dev_cap) dev->caps.port_type[i] = MLX4_PORT_TYPE_IB; else dev->caps.port_type[i] = MLX4_PORT_TYPE_ETH; + dev->caps.possible_type[i] = dev->caps.port_type[i]; + mlx4_priv(dev)->sense.sense_allowed[i] = + dev->caps.supported_type[i] == MLX4_PORT_TYPE_AUTO ? 1 : 0; if (dev->caps.log_num_macs > dev_cap->log_max_macs[i]) { dev->caps.log_num_macs = dev_cap->log_max_macs[i]; @@ -263,7 +265,7 @@ static int mlx4_dev_cap(struct mlx4_dev *dev, struct mlx4_dev_cap *dev_cap) * Change the port configuration of the device. * Every user of this function must hold the port mutex. */ -static int mlx4_change_port_types(struct mlx4_dev *dev, +int mlx4_change_port_types(struct mlx4_dev *dev, enum mlx4_port_type *port_types) { int err = 0; @@ -271,6 +273,8 @@ static int mlx4_change_port_types(struct mlx4_dev *dev, int port; for (port = 0; port < dev->caps.num_ports; port++) { + /* Change the port type only if the new type is different + * from the current, and not set to Auto */ if (port_types[port] != dev->caps.port_type[port + 1]) { change = 1; dev->caps.port_type[port + 1] = port_types[port]; @@ -302,10 +306,17 @@ static ssize_t show_port_type(struct device *dev, struct mlx4_port_info *info = container_of(attr, struct mlx4_port_info, port_attr); struct mlx4_dev *mdev = info->dev; + char type[8]; + + sprintf(type, "%s", + (mdev->caps.port_type[info->port] == MLX4_PORT_TYPE_IB) ? + "ib" : "eth"); + if (mdev->caps.possible_type[info->port] == MLX4_PORT_TYPE_AUTO) + sprintf(buf, "auto (%s)\n", type); + else + sprintf(buf, "%s\n", type); - return sprintf(buf, "%s\n", - mdev->caps.port_type[info->port] == MLX4_PORT_TYPE_IB ? - "ib" : "eth"); + return strlen(buf); } static ssize_t set_port_type(struct device *dev, @@ -317,6 +328,7 @@ static ssize_t set_port_type(struct device *dev, struct mlx4_dev *mdev = info->dev; struct mlx4_priv *priv = mlx4_priv(mdev); enum mlx4_port_type types[MLX4_MAX_PORTS]; + enum mlx4_port_type new_types[MLX4_MAX_PORTS]; int i; int err = 0; @@ -324,24 +336,52 @@ static ssize_t set_port_type(struct device *dev, info->tmp_type = MLX4_PORT_TYPE_IB; else if (!strcmp(buf, "eth\n")) info->tmp_type = MLX4_PORT_TYPE_ETH; + else if (!strcmp(buf, "auto\n")) + info->tmp_type = MLX4_PORT_TYPE_AUTO; else { mlx4_err(mdev, "%s is not supported port type\n", buf); return -EINVAL; } mutex_lock(&priv->port_mutex); - for (i = 0; i < mdev->caps.num_ports; i++) + /* Possible type is always the one that was delivered */ + mdev->caps.possible_type[info->port] = info->tmp_type; + + for (i = 0; i < mdev->caps.num_ports; i++) { types[i] = priv->port[i+1].tmp_type ? priv->port[i+1].tmp_type : - mdev->caps.port_type[i+1]; + mdev->caps.possible_type[i+1]; + if (types[i] == MLX4_PORT_TYPE_AUTO) + types[i] = mdev->caps.port_type[i+1]; + } + + if (!(mdev->caps.flags & MLX4_DEV_CAP_FLAG_DPDP)) { + for (i = 1; i <= mdev->caps.num_ports; i++) { + if (mdev->caps.possible_type[i] == MLX4_PORT_TYPE_AUTO) { + mdev->caps.possible_type[i] = mdev->caps.port_type[i]; + err = -EINVAL; + } + } + } + if (err) { + mlx4_err(mdev, "Auto sensing is not supported on this HCA. " + "Set only 'eth' or 'ib' for both ports " + "(should be the same)\n"); + goto out; + } - err = mlx4_check_port_params(mdev, types); + mlx4_do_sense_ports(mdev, new_types, types); + + err = mlx4_check_port_params(mdev, new_types); if (err) goto out; - for (i = 1; i <= mdev->caps.num_ports; i++) - priv->port[i].tmp_type = 0; + /* We are about to apply the changes after the configuration + * was verified, no need to remember the temporary types + * any more */ + for (i = 0; i < mdev->caps.num_ports; i++) + priv->port[i + 1].tmp_type = 0; - err = mlx4_change_port_types(mdev, types); + err = mlx4_change_port_types(mdev, new_types); out: mutex_unlock(&priv->port_mutex); @@ -1113,14 +1153,21 @@ static int __mlx4_init_one(struct pci_dev *pdev, const struct pci_device_id *id) goto err_port; } - err = mlx4_register_device(dev); + err = mlx4_sense_init(dev); if (err) goto err_port; + err = mlx4_register_device(dev); + if (err) + goto err_sense; + pci_set_drvdata(pdev, dev); return 0; +err_sense: + mlx4_sense_cleanup(dev); + err_port: for (port = 1; port <= dev->caps.num_ports; port++) mlx4_cleanup_port_info(&priv->port[port]); @@ -1182,13 +1229,12 @@ static void mlx4_remove_one(struct pci_dev *pdev) int p; if (dev) { + mlx4_sense_cleanup(dev); mlx4_unregister_device(dev); - for (p = 1; p <= dev->caps.num_ports; p++) { mlx4_cleanup_port_info(&priv->port[p]); mlx4_CLOSE_PORT(dev, p); } - mlx4_cleanup_mcg_table(dev); mlx4_cleanup_qp_table(dev); mlx4_cleanup_srq_table(dev); diff --git a/drivers/net/mlx4/mlx4.h b/drivers/net/mlx4/mlx4.h index e0213ba..347d994 100644 --- a/drivers/net/mlx4/mlx4.h +++ b/drivers/net/mlx4/mlx4.h @@ -40,6 +40,8 @@ #include #include #include +#include +#include #include #include @@ -276,6 +278,15 @@ struct mlx4_port_info { struct mlx4_vlan_table vlan_table; }; +struct mlx4_sense { + struct mlx4_dev *dev; + u8 do_sense_port[MLX4_MAX_PORTS + 1]; + u8 sense_allowed[MLX4_MAX_PORTS + 1]; + struct delayed_work sense_poll; + struct workqueue_struct *sense_wq; + u32 resched; +}; + struct mlx4_priv { struct mlx4_dev dev; @@ -305,6 +316,7 @@ struct mlx4_priv { struct mlx4_uar driver_uar; void __iomem *kar; struct mlx4_port_info port[MLX4_MAX_PORTS + 1]; + struct mlx4_sense sense; struct mutex port_mutex; }; @@ -313,6 +325,8 @@ static inline struct mlx4_priv *mlx4_priv(struct mlx4_dev *dev) return container_of(dev, struct mlx4_priv, dev); } +#define MLX4_SENSE_RANGE (HZ * 3) + u32 mlx4_bitmap_alloc(struct mlx4_bitmap *bitmap); void mlx4_bitmap_free(struct mlx4_bitmap *bitmap, u32 obj); u32 mlx4_bitmap_alloc_range(struct mlx4_bitmap *bitmap, int cnt, int align); @@ -379,6 +393,18 @@ void mlx4_srq_event(struct mlx4_dev *dev, u32 srqn, int event_type); void mlx4_handle_catas_err(struct mlx4_dev *dev); +void mlx4_do_sense_ports(struct mlx4_dev *dev, + enum mlx4_port_type *stype, + enum mlx4_port_type *defaults); +void mlx4_start_sense(struct mlx4_dev *dev); +void mlx4_stop_sense(struct mlx4_dev *dev); +int mlx4_sense_init(struct mlx4_dev *dev); +void mlx4_sense_cleanup(struct mlx4_dev *dev); +int mlx4_check_port_params(struct mlx4_dev *dev, + enum mlx4_port_type *port_type); +int mlx4_change_port_types(struct mlx4_dev *dev, + enum mlx4_port_type *port_types); + void mlx4_init_mac_table(struct mlx4_dev *dev, struct mlx4_mac_table *table); void mlx4_init_vlan_table(struct mlx4_dev *dev, struct mlx4_vlan_table *table); diff --git a/drivers/net/mlx4/sense.c b/drivers/net/mlx4/sense.c new file mode 100644 index 0000000..f3c5411 --- /dev/null +++ b/drivers/net/mlx4/sense.c @@ -0,0 +1,174 @@ +/* + * Copyright (c) 2007 Mellanox Technologies. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ + +#include +#include + +#include + +#include "mlx4.h" + + +static int mlx4_SENSE_PORT(struct mlx4_dev *dev, int port, + enum mlx4_port_type *type) +{ + u64 out_param; + int err = 0; + + err = mlx4_cmd_imm(dev, 0, &out_param, port, 0, + MLX4_CMD_SENSE_PORT, MLX4_CMD_TIME_CLASS_B); + if (err) { + mlx4_err(dev, "Sense command failed for port: %d\n", port); + return err; + } + + if (out_param > 2) { + mlx4_err(dev, "Sense returned illegal value: 0x%llx\n", out_param); + return EINVAL; + } + + *type = out_param; + return 0; +} + +void mlx4_do_sense_ports(struct mlx4_dev *dev, + enum mlx4_port_type *stype, + enum mlx4_port_type *defaults) +{ + struct mlx4_sense *sense = &mlx4_priv(dev)->sense; + int err; + int i; + + for (i = 1; i <= dev->caps.num_ports; i++) { + stype[i-1] = 0; + if (sense->do_sense_port[i] && sense->sense_allowed[i] && + dev->caps.possible_type[i] == MLX4_PORT_TYPE_AUTO) { + err = mlx4_SENSE_PORT(dev, i, &stype[i-1]); + if (err) + stype[i-1] = defaults[i-1]; + } else + stype[i-1] = defaults[i-1]; + } + + /* + * Adjust port configuration: + * If port 1 sensed nothing and port 2 is IB, set both as IB + * If port 2 sensed nothing and port 1 is Eth, set both as Eth + */ + if (stype[0] == MLX4_PORT_TYPE_ETH) { + for (i = 1; i < dev->caps.num_ports; i++) + stype[i] = stype[i] ? stype[i] : MLX4_PORT_TYPE_ETH; + } + if (stype[dev->caps.num_ports - 1] == MLX4_PORT_TYPE_IB) { + for (i = 0; i < dev->caps.num_ports - 1; i++) + stype[i] = stype[i] ? stype[i] : MLX4_PORT_TYPE_IB; + } + + /* + * If sensed nothing, remain in current configuration. + */ + for (i = 0; i < dev->caps.num_ports; i++) + stype[i] = stype[i] ? stype[i] : defaults[i]; + +} + +static void mlx4_sense_port(struct work_struct *work) +{ + struct delayed_work *delay = container_of(work, struct delayed_work, work); + struct mlx4_sense *sense = container_of(delay, struct mlx4_sense, + sense_poll); + struct mlx4_dev *dev = sense->dev; + struct mlx4_priv *priv = mlx4_priv(dev); + enum mlx4_port_type stype[MLX4_MAX_PORTS]; + + mutex_lock(&priv->port_mutex); + mlx4_do_sense_ports(dev, stype, &dev->caps.port_type[1]); + + if (mlx4_check_port_params(dev, stype)) + goto sense_again; + + if (mlx4_change_port_types(dev, stype)) + mlx4_err(dev, "Failed to change port_types\n"); + +sense_again: + mutex_unlock(&priv->port_mutex); + if (sense->resched) + queue_delayed_work(sense->sense_wq , &sense->sense_poll, + round_jiffies(MLX4_SENSE_RANGE)); +} + +void mlx4_start_sense(struct mlx4_dev *dev) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + struct mlx4_sense *sense = &priv->sense; + + if (!(dev->caps.flags & MLX4_DEV_CAP_FLAG_DPDP)) + return; + + sense->resched = 1; + queue_delayed_work(sense->sense_wq , &sense->sense_poll, + round_jiffies(MLX4_SENSE_RANGE)); +} + + +void mlx4_stop_sense(struct mlx4_dev *dev) +{ + mlx4_priv(dev)->sense.resched = 0; +} + +int mlx4_sense_init(struct mlx4_dev *dev) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + struct mlx4_sense *sense = &priv->sense; + int port; + + sense->dev = dev; + sense->sense_wq = create_singlethread_workqueue("mlx4_sense"); + if (!sense->sense_wq) + return -ENOMEM; + + for (port = 1; port <= dev->caps.num_ports; port++) + sense->do_sense_port[port] = 1; + + INIT_DELAYED_WORK_DEFERRABLE(&sense->sense_poll, mlx4_sense_port); + + return 0; +} + +void mlx4_sense_cleanup(struct mlx4_dev *dev) +{ + mlx4_stop_sense(dev); + cancel_delayed_work_sync(&mlx4_priv(dev)->sense.sense_poll); + destroy_workqueue(mlx4_priv(dev)->sense.sense_wq); +} + diff --git a/include/linux/mlx4/cmd.h b/include/linux/mlx4/cmd.h index cf9c679..0f82293 100644 --- a/include/linux/mlx4/cmd.h +++ b/include/linux/mlx4/cmd.h @@ -55,6 +55,7 @@ enum { MLX4_CMD_CLOSE_PORT = 0xa, MLX4_CMD_QUERY_HCA = 0xb, MLX4_CMD_QUERY_PORT = 0x43, + MLX4_CMD_SENSE_PORT = 0x4d, MLX4_CMD_SET_PORT = 0xc, MLX4_CMD_ACCESS_DDR = 0x2e, MLX4_CMD_MAP_ICM = 0xffa, diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index 8f659cc..3aff8a6 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -155,8 +155,9 @@ enum mlx4_qp_region { }; enum mlx4_port_type { - MLX4_PORT_TYPE_IB = 1 << 0, - MLX4_PORT_TYPE_ETH = 1 << 1, + MLX4_PORT_TYPE_IB = 1, + MLX4_PORT_TYPE_ETH = 2, + MLX4_PORT_TYPE_AUTO = 3 }; enum mlx4_special_vlan_idx { @@ -237,6 +238,7 @@ struct mlx4_caps { enum mlx4_port_type port_type[MLX4_MAX_PORTS + 1]; u8 supported_type[MLX4_MAX_PORTS + 1]; u32 port_mask; + enum mlx4_port_type possible_type[MLX4_MAX_PORTS + 1]; }; struct mlx4_buf_list { -- 1.6.1.3 From yevgenyp at mellanox.co.il Tue Mar 3 04:43:10 2009 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Tue, 03 Mar 2009 14:43:10 +0200 Subject: [ofa-general][PATCH 2/2] mlx4_core: Don't perform SET_PORT command for Eth port from core. Message-ID: <49AD25DE.9010501@mellanox.co.il> The same operation is performed when the Ethernet driver initializes the port. If the SET_PORT command is being performed without INIT_PORT command that follows it, the link sensing fsm is found in a bad state. Removing the command from the mlx4_core ensures this situation never happens. Signed-off-by: Yevgeny Petrilin --- drivers/net/mlx4/port.c | 13 +++++-------- 1 files changed, 5 insertions(+), 8 deletions(-) diff --git a/drivers/net/mlx4/port.c b/drivers/net/mlx4/port.c index 0a057e5..7cce334 100644 --- a/drivers/net/mlx4/port.c +++ b/drivers/net/mlx4/port.c @@ -298,20 +298,17 @@ int mlx4_SET_PORT(struct mlx4_dev *dev, u8 port) { struct mlx4_cmd_mailbox *mailbox; int err; - u8 is_eth = dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH; mailbox = mlx4_alloc_cmd_mailbox(dev); if (IS_ERR(mailbox)) return PTR_ERR(mailbox); memset(mailbox->buf, 0, 256); - if (is_eth) { - ((u8 *) mailbox->buf)[3] = 6; - ((__be16 *) mailbox->buf)[4] = cpu_to_be16(1 << 15); - ((__be16 *) mailbox->buf)[6] = cpu_to_be16(1 << 15); - } else - ((__be32 *) mailbox->buf)[1] = dev->caps.ib_port_def_cap[port]; - err = mlx4_cmd(dev, mailbox->dma, port, is_eth, MLX4_CMD_SET_PORT, + if (dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH) + return 0; + + ((__be32 *) mailbox->buf)[1] = dev->caps.ib_port_def_cap[port]; + err = mlx4_cmd(dev, mailbox->dma, port, 0, MLX4_CMD_SET_PORT, MLX4_CMD_TIME_CLASS_B); mlx4_free_cmd_mailbox(dev, mailbox); -- 1.6.1.3 From roel.kluin at gmail.com Tue Mar 3 05:38:35 2009 From: roel.kluin at gmail.com (Roel Kluin) Date: Tue, 03 Mar 2009 14:38:35 +0100 Subject: [ofa-general] ***SPAM*** [PATCH] RDMA/cxgb3: test before subtraction on unsigned Message-ID: <49AD32DB.20407@gmail.com> I think there's someting wrong in iwch_post_send(): // vi drivers/infiniband/hw/cxgb3/iwch_qp.c +353 int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, struct ib_send_wr **bad_wr) { ... u32 num_wrs; ... num_wrs = Q_FREECNT(qhp->wq.sq_rptr, qhp->wq.sq_wptr, qhp->wq.sq_size_log2); if (num_wrs <= 0) { spin_unlock_irqrestore(&qhp->lock, flag); return -ENOMEM; } // vi drivers/infiniband/hw/cxgb3/cxio_wr.h +50 #define Q_FREECNT(rptr,wptr,size_log2) ((1UL<8-------------8<--------------------------------- num_wrs is unsigned so test before the subtraction. Signed-off-by: Roel Kluin --- diff --git a/drivers/infiniband/hw/cxgb3/iwch_qp.c b/drivers/infiniband/hw/cxgb3/iwch_qp.c index 19661b2..1c6ebaf 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_qp.c +++ b/drivers/infiniband/hw/cxgb3/iwch_qp.c @@ -369,12 +369,13 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, spin_unlock_irqrestore(&qhp->lock, flag); return -EINVAL; } - num_wrs = Q_FREECNT(qhp->wq.sq_rptr, qhp->wq.sq_wptr, - qhp->wq.sq_size_log2); - if (num_wrs <= 0) { + if ((1UL << qhp->wq.sq_size_log2) + qhp->wq.sq_rptr <= + qhp->wq.sq_wptr) { spin_unlock_irqrestore(&qhp->lock, flag); return -ENOMEM; } + num_wrs = Q_FREECNT(qhp->wq.sq_rptr, qhp->wq.sq_wptr, + qhp->wq.sq_size_log2); while (wr) { if (num_wrs == 0) { err = -ENOMEM; From hal.rosenstock at gmail.com Tue Mar 3 05:55:15 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 3 Mar 2009 08:55:15 -0500 Subject: ***SPAM*** Re: [ofa-general] [PATCH] IB/core: ib_post_send_mad() returns zero but doesn't generate send completion In-Reply-To: <1236041844.3948.293.camel@chromite.mv.qlogic.com> References: <1235771157.3948.233.camel@chromite.mv.qlogic.com> <1236041844.3948.293.camel@chromite.mv.qlogic.com> Message-ID: On Mon, Mar 2, 2009 at 7:57 PM, Ralph Campbell wrote: > On Sun, 2009-03-01 at 17:23 -0800, Hal Rosenstock wrote: >> On Fri, Feb 27, 2009 at 4:45 PM, Ralph Campbell >> wrote: >> >> Good catches. >> >> > If ib_post_send_mad() returns zero, it guarantees that there will be >> > a callback to the send_buf->mad_agent->send_handler() so that the >> > sender can call ib_free_send_mad(). Otherwise, the ib_mad_send_buf >> > will be leaked and the mad_agent reference count will never go to zero >> > and the IB device module cannot be unloaded. >> > The above can happen without this patch if process_mad() returns >> > (IB_MAD_RESULT_SUCCESS | IB_MAD_RESULT_CONSUMED). >> >> This looks right although I'm not sure why this wasn't seen a long time ago. > > mthca returns IB_MAD_RESULT_CONSUMED for TRAP repress. Also, locally generated traps (which recent OpenSMs use for SMAs not generating SM priority change traps). > I verified that using libibumad to send a direct routed TRAP repress with hop=1/cnt=0 > does hang on mthca. I think opensm only sends LID routed TRAP repress > so that would explain why this wasn't seen earlier. AFAIK OpenSM responds with trap repress based on the style of the incoming trap (e.g. LID or direct routed). > For ipath, it was fairly recently that we made SubnSet(Portinfo) to > disabled to not send a reply and we have a program that uses libibumad > to disable ports. Without the patch, the program hangs. > >> > If process_mad() returns IB_MAD_RESULT_SUCCESS and there is no agent >> > registered to receive the mad being sent, handle_outgoing_dr_smp() >> > returns zero which causes a MAD packet which is at the end of the >> > directed route to be incorrectly sent on the wire but doesn't cause >> > a hang since the HCA generates a send completion. >> >> This also looks right. >> >> Does the packet really get out on the IB wire ? Was that with a >> Mellanox HCA ? I don't recall ever seeing these extra packets but >> maybe they end up swallowed by certain HCAs. > > Yes, a packet is sent with either mthca or ipath. What does it look like ? Is it responded to ? -- Hal >> Unfortunately, I have no current way of regressing these changes. I'm >> presuming you've done this with Mellanox HCAs and with OpenSM running. > > opensm runs fine with the patch applied on either mthca or ipath. > >> One other question is whether this should be separated into two increments. > > I think the changes are small enough and related enough to justify a > single patch but I will split it into two if someone feels I should. > - Show quoted text - >> -- Hal >> >> > Signed-off-by: Ralph Campbell >> > >> > diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c >> > index dbcd285..62a99dc 100644 >> > --- a/drivers/infiniband/core/mad.c >> > +++ b/drivers/infiniband/core/mad.c >> > @@ -742,9 +742,7 @@ static int handle_outgoing_dr_smp(struct ib_mad_agent_private *mad_agent_priv, >> >                break; >> >        case IB_MAD_RESULT_SUCCESS | IB_MAD_RESULT_CONSUMED: >> >                kmem_cache_free(ib_mad_cache, mad_priv); >> > -               kfree(local); >> > -               ret = 1; >> > -               goto out; >> > +               break; >> >        case IB_MAD_RESULT_SUCCESS: >> >                /* Treat like an incoming receive MAD */ >> >                port_priv = ib_get_mad_port(mad_agent_priv->agent.device, >> > @@ -755,10 +753,12 @@ static int handle_outgoing_dr_smp(struct ib_mad_agent_private *mad_agent_priv, >> >                                                        &mad_priv->mad.mad); >> >                } >> >                if (!port_priv || !recv_mad_agent) { >> > +                       /* >> > +                        * No receiving agent so drop packet and >> > +                        * generate send completion. >> > +                        */ >> >                        kmem_cache_free(ib_mad_cache, mad_priv); >> > -                       kfree(local); >> > -                       ret = 0; >> > -                       goto out; >> > +                       break; >> >                } >> >                local->mad_priv = mad_priv; >> >                local->recv_mad_agent = recv_mad_agent; >> > >> > >> > _______________________________________________ >> > general mailing list >> > general at lists.openfabrics.org >> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> > >> > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >> > > > From hnrose at comcast.net Tue Mar 3 07:26:44 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 3 Mar 2009 10:26:44 -0500 Subject: [ofa-general] ***SPAM*** [PATCH][TRIVIAL] ibsim/sim_mad.c: Eliminate a couple of unneeded blank lines Message-ID: <20090303152644.GC4425@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c index 740e671..d73ce25 100644 --- a/ibsim/sim_mad.c +++ b/ibsim/sim_mad.c @@ -86,7 +86,6 @@ static Smpfn *attrs[IB_PERFORMANCE_CLASS + 1][0xff] = { [IB_GSI_PORT_COUNTERS_EXT] = do_extcounters, [IB_GSI_ATTR_LAST] 0, - }, }; @@ -95,7 +94,6 @@ static EncodeTrapfn *encodetrap[] = { [TRAP_144] encode_trap144, [TRAP_NUM_LAST] 0, - }; extern Node *nodes; From hnrose at comcast.net Tue Mar 3 07:25:58 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 3 Mar 2009 10:25:58 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] ibsim/sim_net.c: Eliminate IsLEDInfoSupported PortInfo CapabilityMask bit Message-ID: <20090303152558.GB4425@comcast.net> rather than implement LEDInfo attribute support in the simulator Signed-off-by: Hal Rosenstock --- diff --git a/ibsim/sim_net.c b/ibsim/sim_net.c index 223bc10..be5ceab 100644 --- a/ibsim/sim_net.c +++ b/ibsim/sim_net.c @@ -67,7 +67,7 @@ Port *default_port; static const uint8_t smaport[] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFE, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x02, 0x48, + 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x48, 0x00, 0x00, 0x0F, 0xF9, 0x00, 0x03, 0x03, 0x01, 0x14, 0x52, 0x00, 0x11, 0x10, 0x40, 0x00, 0x08, 0x08, 0x03, 0x00, 0x40, 0x00, 0x00, 0x00, 0x00, @@ -100,7 +100,7 @@ static const uint8_t swport_down[] = { static const uint8_t hcaport[] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFE, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x02, 0x00, 0x01, 0x00, 0x50, 0x02, 0x48, + 0x00, 0x02, 0x00, 0x01, 0x00, 0x50, 0x00, 0x48, 0x00, 0x00, 0x0F, 0xF9, 0x01, 0x03, 0x03, 0x02, 0x12, 0x52, 0x00, 0x11, 0x40, 0x40, 0x00, 0x08, 0x08, 0x04, 0xFF, 0x10, 0x00, 0x00, 0x00, 0x00, From hnrose at comcast.net Tue Mar 3 07:25:08 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 3 Mar 2009 10:25:08 -0500 Subject: [ofa-general] ***SPAM*** [PATCH][TRIVIAL] opensm/osm_trap_rcv.c: Cosmetic changes Message-ID: <20090303152508.GA4425@comcast.net> Removed extraneous parentheses Eliminated unneeded __ and __osm_ prefixes to static routine names Fixed some commentary typos Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_trap_rcv.c b/opensm/opensm/osm_trap_rcv.c index 46fbad7..a5491e1 100644 --- a/opensm/opensm/osm_trap_rcv.c +++ b/opensm/opensm/osm_trap_rcv.c @@ -72,7 +72,7 @@ extern void osm_req_get_node_desc(IN osm_sm_t * sm, osm_physp_t *p_physp); * * A timer running in the background will toggle a timer counter * that should be referenced by the aging algorithm. - * To provide an efficient handling of aging. We also track all traps + * To provide an efficient handling of aging, we also track all traps * in a sorted list by their aging. * * The generic Aging Tracker mechanism is implemented in the @@ -156,12 +156,12 @@ osm_trap_rcv_aging_tracker_callback(IN uint64_t key, #define CRC32_POLYNOMIAL 0xEDB88320L /* calculate the crc for a given buffer */ -static uint32_t __osm_trap_calc_crc32(void *buffer, uint32_t count) +static uint32_t trap_calc_crc32(void *buffer, uint32_t count) { uint32_t temp1, temp2; uint32_t crc = -1L; unsigned char *p = (unsigned char *)buffer; - /* pre - calculated table for faster crc calculation */ + /* precalculated table for faster crc calculation */ static uint32_t crc_table[256]; static boolean_t first = TRUE; int i, j; @@ -173,7 +173,7 @@ static uint32_t __osm_trap_calc_crc32(void *buffer, uint32_t count) crc = i; for (j = 8; j > 0; j--) if (crc & 1) - crc = (crc >> 1) ^ CRC32_POLYNOMIAL; + crc = crc >> 1 ^ CRC32_POLYNOMIAL; else crc >>= 1; crc_table[i] = crc; @@ -200,21 +200,20 @@ static uint32_t __osm_trap_calc_crc32(void *buffer, uint32_t count) 16b 16b 32b */ static void -__osm_trap_get_key(IN uint16_t lid, - IN uint8_t port_num, - IN ib_mad_notice_attr_t * p_ntci, OUT uint64_t * trap_key) +trap_get_key(IN uint16_t lid, IN uint8_t port_num, + IN ib_mad_notice_attr_t * p_ntci, OUT uint64_t * trap_key) { uint32_t crc = 0; CL_ASSERT(trap_key); - crc = __osm_trap_calc_crc32(p_ntci, sizeof(ib_mad_notice_attr_t)); + crc = trap_calc_crc32(p_ntci, sizeof(ib_mad_notice_attr_t)); *trap_key = ((uint64_t) port_num << 48) | ((uint64_t) lid << 32) | crc; } /********************************************************************** **********************************************************************/ -static int __print_num_received(IN uint32_t num_received) +static int print_num_received(IN uint32_t num_received) { uint32_t i; @@ -278,8 +277,8 @@ static int disable_port(osm_sm_t *sm, osm_physp_t *p) /********************************************************************** **********************************************************************/ static void -__osm_trap_rcv_process_request(IN osm_sm_t * sm, - IN const osm_madw_t * const p_madw) +trap_rcv_process_request(IN osm_sm_t * sm, + IN const osm_madw_t * const p_madw) { uint8_t payload[sizeof(ib_mad_notice_attr_t)]; ib_smp_t *p_smp; @@ -333,13 +332,13 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, * payload. */ - memcpy(payload, &(p_smp->data), IB_SMP_DATA_SIZE); + memcpy(payload, &p_smp->data, IB_SMP_DATA_SIZE); memcpy(&tmp_madw, p_madw, sizeof(tmp_madw)); if (is_gsi == FALSE) { /* We are in smi flow */ /* - * When we received a TRAP with dlid = 0 - it means it + * When we receive a TRAP with dlid = 0 - it means it * came from our own node. So we need to fix it. */ @@ -438,9 +437,9 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, if (is_gsi == FALSE) { if (ib_notice_is_generic(p_ntci) && - ((p_ntci->g_or_v.generic.trap_num == CL_HTON16(129)) || - (p_ntci->g_or_v.generic.trap_num == CL_HTON16(130)) || - (p_ntci->g_or_v.generic.trap_num == CL_HTON16(131)))) { + (p_ntci->g_or_v.generic.trap_num == CL_HTON16(129) || + p_ntci->g_or_v.generic.trap_num == CL_HTON16(130) || + p_ntci->g_or_v.generic.trap_num == CL_HTON16(131))) { /* If this is a trap 129, 130, or 131 - then this is a * trap signaling a change on a physical port. * Mark the physp_change_trap flag as TRUE. @@ -454,10 +453,9 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, If not - the port_number in the key will be zero. */ if (physp_change_trap == TRUE) { port_num = p_ntci->data_details.ntc_129_131.port_num; - __osm_trap_get_key(source_lid, port_num, p_ntci, - &trap_key); + trap_get_key(source_lid, port_num, p_ntci, &trap_key); } else - __osm_trap_get_key(source_lid, 0, p_ntci, &trap_key); + trap_get_key(source_lid, 0, p_ntci, &trap_key); /* try to find it in the aging tracker */ num_received = @@ -466,7 +464,7 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, /* Now we know how many times it provided this trap */ if (num_received > 10) { - if (__print_num_received(num_received)) + if (print_num_received(num_received)) OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3804: " "Received trap %u times consecutively\n", num_received); @@ -540,7 +538,7 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, /* If was already registered do nothing more */ if (num_received > 10 && run_heavy_sweep == FALSE) { - if (__print_num_received(num_received)) + if (print_num_received(num_received)) OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, "Continuously received this trap %u times. Ignoring\n", num_received); @@ -549,9 +547,8 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, } /* Check for node description update. IB Spec v1.2.1 pg 823 */ - if ((p_ntci->data_details.ntc_144.local_changes & TRAP_144_MASK_OTHER_LOCAL_CHANGES) && - (p_ntci->data_details.ntc_144.change_flgs & TRAP_144_MASK_NODE_DESCRIPTION_CHANGE) - ) { + if (p_ntci->data_details.ntc_144.local_changes & TRAP_144_MASK_OTHER_LOCAL_CHANGES && + p_ntci->data_details.ntc_144.change_flgs & TRAP_144_MASK_NODE_DESCRIPTION_CHANGE) { OSM_LOG(sm->p_log, OSM_LOG_INFO, "Trap 144 Node description update\n"); if (p_physp) { @@ -574,9 +571,9 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, TODO: In the future we can change this to just getting PortInfo on this port instead of sweeping the entire subnet. */ if (ib_notice_is_generic(p_ntci) && - ((cl_ntoh16(p_ntci->g_or_v.generic.trap_num) == 128) || - (cl_ntoh16(p_ntci->g_or_v.generic.trap_num) == 144) || - (cl_ntoh16(p_ntci->g_or_v.generic.trap_num) == 145) || + (cl_ntoh16(p_ntci->g_or_v.generic.trap_num) == 128 || + cl_ntoh16(p_ntci->g_or_v.generic.trap_num) == 144 || + cl_ntoh16(p_ntci->g_or_v.generic.trap_num) == 145 || run_heavy_sweep)) { OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, "Forcing heavy sweep. Received trap:%u\n", @@ -603,8 +600,8 @@ __osm_trap_rcv_process_request(IN osm_sm_t * sm, "Cannot update issuer_gid!\n"); goto Exit; } - memcpy(&(p_ntci->issuer_gid), - &(tmp_madw.mad_addr.addr_type.gsi.grh_info.src_gid), + memcpy(&p_ntci->issuer_gid, + &tmp_madw.mad_addr.addr_type.gsi.grh_info.src_gid, sizeof(ib_gid_t)); } else { /* Need to use the IssuerLID */ @@ -651,31 +648,12 @@ Exit: OSM_LOG_EXIT(sm->p_log); } -#if 0 -/********************************************************************** - CURRENTLY WE ARE NOT CREATING TRAPS - SO THIS CALL IS AN ERROR -**********************************************************************/ -static void -__osm_trap_rcv_process_sm(IN osm_sm_t * sm, - IN const osm_remote_sm_t * const p_sm) -{ - /* const ib_sm_info_t* p_smi; */ - - OSM_LOG_ENTER(sm->p_log); - - OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3807: " - "This function is not supported yet\n"); - - OSM_LOG_EXIT(sm->p_log); -} -#endif - /********************************************************************** CURRENTLY WE ARE NOT CREATING TRAPS - SO THIS CALL IN AN ERROR **********************************************************************/ static void -__osm_trap_rcv_process_response(IN osm_sm_t * sm, - IN const osm_madw_t * const p_madw) +trap_rcv_process_response(IN osm_sm_t * sm, + IN const osm_madw_t * const p_madw) { OSM_LOG_ENTER(sm->p_log); @@ -706,9 +684,9 @@ void osm_trap_rcv_process(IN void *context, IN void *data) SM's Trap. */ if (ib_smp_is_response(p_smp)) - __osm_trap_rcv_process_response(sm, p_madw); + trap_rcv_process_response(sm, p_madw); else - __osm_trap_rcv_process_request(sm, p_madw); + trap_rcv_process_request(sm, p_madw); OSM_LOG_EXIT(sm->p_log); } From swise at opengridcomputing.com Tue Mar 3 07:41:56 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Tue, 03 Mar 2009 09:41:56 -0600 Subject: [ofa-general] Re: [PATCH] RDMA/cxgb3: test before subtraction on unsigned In-Reply-To: <49AD32DB.20407@gmail.com> References: <49AD32DB.20407@gmail.com> Message-ID: <49AD4FC4.2020100@opengridcomputing.com> Roel Kluin wrote: > I think there's someting wrong in iwch_post_send(): > > // vi drivers/infiniband/hw/cxgb3/iwch_qp.c +353 > int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, > struct ib_send_wr **bad_wr) > { > ... > u32 num_wrs; > ... > num_wrs = Q_FREECNT(qhp->wq.sq_rptr, qhp->wq.sq_wptr, > qhp->wq.sq_size_log2); > if (num_wrs <= 0) { > spin_unlock_irqrestore(&qhp->lock, flag); > return -ENOMEM; > } > > // vi drivers/infiniband/hw/cxgb3/cxio_wr.h +50 > #define Q_FREECNT(rptr,wptr,size_log2) ((1UL< > Since num_wrs is unsigned, I think the test should occur before the subtraction, > right? please review. > ------------------------------>8-------------8<--------------------------------- > num_wrs is unsigned so test before the subtraction. > > Signed-off-by: Roel Kluin > --- > diff --git a/drivers/infiniband/hw/cxgb3/iwch_qp.c b/drivers/infiniband/hw/cxgb3/iwch_qp.c > index 19661b2..1c6ebaf 100644 > --- a/drivers/infiniband/hw/cxgb3/iwch_qp.c > +++ b/drivers/infiniband/hw/cxgb3/iwch_qp.c > @@ -369,12 +369,13 @@ int iwch_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr, > spin_unlock_irqrestore(&qhp->lock, flag); > return -EINVAL; > } > - num_wrs = Q_FREECNT(qhp->wq.sq_rptr, qhp->wq.sq_wptr, > - qhp->wq.sq_size_log2); > - if (num_wrs <= 0) { > + if ((1UL << qhp->wq.sq_size_log2) + qhp->wq.sq_rptr <= > + qhp->wq.sq_wptr) { > spin_unlock_irqrestore(&qhp->lock, flag); > return -ENOMEM; > } > + num_wrs = Q_FREECNT(qhp->wq.sq_rptr, qhp->wq.sq_wptr, > + qhp->wq.sq_size_log2); > while (wr) { > if (num_wrs == 0) { > err = -ENOMEM; > > I think the test is just wrong. It should be: if (num_wrs == 0). Q_FREECNT() will not return < 0. The free count is either zero if it is full, or non zero count of the number of free elements. Steve. From nicolas.morey-chaisemartin at ext.bull.net Tue Mar 3 07:52:28 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Tue, 03 Mar 2009 16:52:28 +0100 Subject: [ofa-general] [PATCH] opensm/osm_ucast_ftree.c: Removed useless initialisation on switch indexes Message-ID: <49AD523C.2010501@ext.bull.net> Signed-off-by: Nicolas Morey-Chaisemartin --- There is no need to initialize these values to 0 as the struct has been memset to 0 after its allocation. opensm/opensm/osm_ucast_ftree.c | 4 ---- 1 files changed, 0 insertions(+), 4 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index d92265b..6e2e7de 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -561,14 +561,10 @@ static ftree_sw_t *__osm_ftree_sw_create(IN ftree_fabric_t * p_ftree, sizeof(ftree_port_group_t *)); if (!p_sw->down_port_groups || !p_sw->up_port_groups) return NULL; - p_sw->down_port_groups_num = 0; - p_sw->up_port_groups_num = 0; /* initialize lft buffer */ memset(p_osm_sw->new_lft, OSM_NO_PATH, IB_LID_UCAST_END_HO + 1); - p_sw->down_port_groups_idx = 0; - return p_sw; } /* __osm_ftree_sw_create() */ -- 1.6.2-rc2.GIT From hnrose at comcast.net Tue Mar 3 08:48:06 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 3 Mar 2009 11:48:06 -0500 Subject: [ofa-general] ***SPAM*** [PATCHv2] ibsim/sim_net.c: Eliminate IsLEDInfoSupported PortInfo CapabilityMask bit Message-ID: <20090303164806.GA9050@comcast.net> rather than implement LEDInfo attribute support in the simulator Signed-off-by: Hal Rosenstock --- diff --git a/ibsim/sim_net.c b/ibsim/sim_net.c index fa05c35..aa91103 100644 --- a/ibsim/sim_net.c +++ b/ibsim/sim_net.c @@ -67,7 +67,7 @@ Port *default_port; static const uint8_t smaport[] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFE, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x02, 0x48, + 0x00, 0x01, 0x00, 0x01, 0x00, 0x00, 0x00, 0x48, 0x00, 0x00, 0x0F, 0xF9, 0x00, 0x03, 0x03, 0x01, 0x14, 0x52, 0x00, 0x11, 0x10, 0x40, 0x00, 0x08, 0x08, 0x03, 0x00, 0x40, 0x00, 0x00, 0x00, 0x00, @@ -100,7 +100,7 @@ static const uint8_t swport_down[] = { static const uint8_t hcaport[] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFE, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x02, 0x00, 0x01, 0x00, 0x50, 0x02, 0x48, + 0x00, 0x02, 0x00, 0x01, 0x00, 0x50, 0x00, 0x48, 0x00, 0x00, 0x0F, 0xF9, 0x01, 0x03, 0x03, 0x02, 0x12, 0x52, 0x00, 0x11, 0x40, 0x40, 0x00, 0x08, 0x08, 0x04, 0xFF, 0x10, 0x00, 0x00, 0x00, 0x00, @@ -111,7 +111,7 @@ static const uint8_t hcaport[] = { static const uint8_t hcaport_down[] = { 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFE, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, - 0x00, 0x02, 0x00, 0x01, 0x00, 0x10, 0x02, 0x48, + 0x00, 0x02, 0x00, 0x01, 0x00, 0x10, 0x00, 0x48, 0x00, 0x00, 0x0F, 0xF9, 0x01, 0x03, 0x03, 0x01, 0x11, 0x22, 0x00, 0x11, 0x40, 0x40, 0x00, 0x08, 0x08, 0x04, 0xE9, 0x10, 0x00, 0x00, 0x00, 0x00, From arlin.r.davis at intel.com Tue Mar 3 10:07:03 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Tue, 3 Mar 2009 10:07:03 -0800 Subject: [ofa-general] Bandwidth of performance with multirail IB In-Reply-To: <49A7885B.3010005@cs.anu.edu.au> References: <20090223211155.730AFE28137@openfabrics.org> <49A378BC.5010806@cs.anu.edu.au> <200902240941.58634.cap@nsc.liu.se> <49A7885B.3010005@cs.anu.edu.au> Message-ID: Jie, >A question on implementation multi-rail with uDAPL connections. > >What I did is open 2 IAs (corresponding to the 2 ports on >HCAs) on each >node. >Then create one EP for each IA, and connect those EPs to the >corresponding >EP at other node. > >Then data been transferred via both EP-connections. Did you verify data flow is across independent HCA channels using perfquery? > >I have been notice that there's a MULTIPATH connection flag for dapl, >but I did >not use it. What's the use of it? Multipath is not supported at this time. I will roll up a uDAPL test and see if I can verify performance. -arlin From ralph.campbell at qlogic.com Tue Mar 3 10:24:02 2009 From: ralph.campbell at qlogic.com (Ralph Campbell) Date: Tue, 03 Mar 2009 10:24:02 -0800 Subject: [ofa-general] [PATCH] IB/core: ib_post_send_mad() returns zero but doesn't generate send completion In-Reply-To: References: <1235771157.3948.233.camel@chromite.mv.qlogic.com> <1236041844.3948.293.camel@chromite.mv.qlogic.com> Message-ID: <1236104642.3948.303.camel@chromite.mv.qlogic.com> On Tue, 2009-03-03 at 05:55 -0800, Hal Rosenstock wrote: > >> Does the packet really get out on the IB wire ? Was that with a > >> Mellanox HCA ? I don't recall ever seeing these extra packets but > >> maybe they end up swallowed by certain HCAs. > > > > Yes, a packet is sent with either mthca or ipath. > > What does it look like ? Is it responded to ? I changed smpquery to send a node info MAD with the method set to SEND so mthca_process_mad would return zero and no matching mad agent receiver. This is what a second node sees & replies. Madeye:recv SMP MAD version....0x1 Class..........0x81 (Directed route SMP) Class version..0x1 Method.........0x3 (Send) Status.........0x00 Hop pointer....0x1 Hop counter....0x0 Trans ID.......0x150c798f86 Attr ID........0x11 (node info) Attr modifier..0x0000 Mkey...........0x0 DR SLID........0xffff DR DLID........0xffff Madeye:sent SMP MAD version....0x1 Class..........0x81 (Directed route SMP) Class version..0x1 Method.........0x81 (Get response) Status.........0x8008 method not supported Hop pointer....0x0 Hop counter....0x0 Trans ID.......0x150c798f86 Attr ID........0x11 (node info) Attr modifier..0x0000 Mkey...........0x0 DR SLID........0xffff DR DLID........0xffff From hnrose at comcast.net Tue Mar 3 11:03:04 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 3 Mar 2009 14:03:04 -0500 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/libvendor/osm_vendor_ibumad.c: Commentary changes Message-ID: <20090303190304.GA14633@comcast.net> Also, formatting change Signed-off-by: Hal Rosenstock --- diff --git a/opensm/libvendor/osm_vendor_ibumad.c b/opensm/libvendor/osm_vendor_ibumad.c index 734a860..11360a8 100644 --- a/opensm/libvendor/osm_vendor_ibumad.c +++ b/opensm/libvendor/osm_vendor_ibumad.c @@ -1033,14 +1033,14 @@ osm_vendor_send(IN osm_bind_handle_t h_bind, umad_set_grh(p_vw->umad, 0); goto Resp; } - /* GSI classes */ + /* GS classes */ umad_set_addr_net(p_vw->umad, p_mad_addr->dest_lid, p_mad_addr->addr_type.gsi.remote_qp, p_mad_addr->addr_type.gsi.service_level, IB_QP1_WELL_KNOWN_Q_KEY); umad_set_grh(p_vw->umad, 0); /* FIXME: GRH support */ umad_set_pkey(p_vw->umad, p_mad_addr->addr_type.gsi.pkey_ix); - if (ib_class_is_rmpp(p_mad->mgmt_class)) { /* RMPP GSI classes FIXME: no GRH */ + if (ib_class_is_rmpp(p_mad->mgmt_class)) { /* RMPP GS classes FIXME: no GRH */ if (!ib_rmpp_is_flag_set((ib_rmpp_mad_t *) p_sa, IB_RMPP_FLAG_ACTIVE)) { /* Clear RMPP header when RMPP not ACTIVE */ @@ -1051,10 +1051,10 @@ osm_vendor_send(IN osm_bind_handle_t h_bind, #ifdef VENDOR_RMPP_SUPPORT } else is_rmpp = 1; - OSM_LOG(p_vend->p_log, OSM_LOG_VERBOSE, "RMPP %d length %d\n", - ib_rmpp_is_flag_set((ib_rmpp_mad_t *) p_sa, - IB_RMPP_FLAG_ACTIVE), - p_madw->mad_size); + OSM_LOG(p_vend->p_log, OSM_LOG_VERBOSE, "RMPP %d length %d\n", + ib_rmpp_is_flag_set((ib_rmpp_mad_t *) p_sa, + IB_RMPP_FLAG_ACTIVE), + p_madw->mad_size); #else } else { p_sa->rmpp_version = 1; From sean.hefty at intel.com Tue Mar 3 11:13:46 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 3 Mar 2009 11:13:46 -0800 Subject: [ofa-general] complib and cl_nodenamemap Message-ID: <49E9FBB3AEB64CEDA3577778C69E6392@amr.corp.intel.com> Sasha, I'd like to see about doing 'something' with nodenamemap. Currently, it's part of complib, but the interfaces and operation are not similar to other complib type abstractions. >From what I can tell, nodenamemap performs two abstractions. At a lower level, it implements functionality similar to cl_map. The main difference is that cl_map requires the user to allocate space for the object being inserted into the map. If we add a new call to complib, we can provide an abstraction of the functionality that nodenamemap requires: cl_map_insert_copy(cl_map_t *p_map, uint64_t key, void *p_object, size_t object_size); There would be internal changes needed to free the object, plus changes to cl_map_remove return values for copied objects. At a higher level, nodenamemap parses an application specific file and stores the contents of the file into a map. I'm not sure that this functionality really belongs as a part of complib. But regardless, it makes more sense to me to separate the parsing of the file from maintaining the data in a map. There's not a need to abstract opening or closing the file. So the only functionality that's needed is parsing a line of input from the file. Each application could link in the necessary code directly, the parsing code could go into a new 'ibcommon' library (not really worth it for 1 call), or the file format just needs to be generic enough to work with a wide variety of applications (and maybe it already is). Is it acceptable to change or remove the nodenamemap abstraction from complib, and if so, is there a specific direction to take? - Sean From hnrose at comcast.net Tue Mar 3 11:12:17 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 3 Mar 2009 14:12:17 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] ibsim/sim_mad.c: Better error handling in send_trap Message-ID: <20090303191217.GA14839@comcast.net> In send_trap, validate trap number prior to determining trap function Made trapnum parameter to send_trap unsigned Also, removed unneeded enum init of NO_NODE in NODE_TYPES Signed-off-by: Hal Rosenstock --- diff --git a/ibsim/sim.h b/ibsim/sim.h index f0e1e16..09812ba 100644 --- a/ibsim/sim.h +++ b/ibsim/sim.h @@ -72,7 +72,7 @@ #define MAXHOPS 16 enum NODE_TYPES { - NO_NODE = 0, + NO_NODE, HCA_NODE, SWITCH_NODE, ROUTER_NODE, @@ -289,7 +289,7 @@ void *sim_cmd_thread(void *file); // sim_mad.c int process_packet(Client * cl, void *p, int size, Client ** dcl); -int send_trap(Port * port, int trapnum); +int send_trap(Port * port, unsigned trapnum); extern Port *default_port; extern int simverb; diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c index d73ce25..32a667e 100644 --- a/ibsim/sim_mad.c +++ b/ibsim/sim_mad.c @@ -1163,7 +1163,7 @@ int process_packet(Client * cl, void *p, int size, Client ** dcl) return -1; if (rpc.method == 0x7) { - IBWARN("got trap repress - drop"); + IBWARN("lid %u got trap repress - dropping", ntohs(r->dlid)); *dcl = 0; return 0; } @@ -1280,20 +1280,21 @@ static int encode_trap_header(char *buf) return 0; } -int send_trap(Port * port, int trapnum) +int send_trap(Port * port, unsigned trapnum) { struct sim_request req; Client *cl; int ret, lid = port->lid; char *data = req.mad + 64; /* data offset */ - EncodeTrapfn *encode_trapfn = encodetrap[trapnum]; + EncodeTrapfn *encode_trapfn; Port *destport; - if (!encode_trapfn) { + if (trapnum >= TRAP_NUM_LAST) { IBWARN("trap number %d not supported", trapnum); return -1; } + encode_trapfn = encodetrap[trapnum]; memset(req.mad, 0, sizeof(req.mad)); encode_trap_header(req.mad); if (encode_trapfn(port, data) < 0) From arlin.r.davis at intel.com Tue Mar 3 12:50:48 2009 From: arlin.r.davis at intel.com (Arlin Davis) Date: Tue, 3 Mar 2009 12:50:48 -0800 Subject: [ofa-general] [PATCH] uDAPL v2 - Add 64 bit counters for IA, EP, and EVD's Message-ID: <000401c99c41$bb937420$7b97070a@amr.corp.intel.com> Add full set of 64-bit counters to uDAPL common code, cma, and scm providers. Replaced the current global set of counters. Details: Use -DDAPL_COUNTERS to build-in counters for cma and scm providers. New extension calls in dat_ib_extensions.h for counters dat_print_counters, dat_query_counters Added counters for operations, async errors, and data Update dtestx (-p) with print and query counter examples Signed-off-by: Arlin Davis --- dapl/common/dapl_debug.c | 202 +++++++++++++++++++++----- dapl/common/dapl_ep_connect.c | 2 +- dapl/common/dapl_ep_create.c | 2 +- dapl/common/dapl_ep_disconnect.c | 2 +- dapl/common/dapl_ep_free.c | 2 +- dapl/common/dapl_ep_get_status.c | 4 +- dapl/common/dapl_ep_post_rdma_read.c | 1 - dapl/common/dapl_ep_post_rdma_write.c | 1 - dapl/common/dapl_ep_post_recv.c | 1 - dapl/common/dapl_ep_post_send.c | 1 - dapl/common/dapl_ep_util.c | 15 ++ dapl/common/dapl_evd_connection_callb.c | 3 +- dapl/common/dapl_evd_cq_async_error_callb.c | 1 + dapl/common/dapl_evd_dequeue.c | 9 +- dapl/common/dapl_evd_dto_callb.c | 2 +- dapl/common/dapl_evd_free.c | 3 +- dapl/common/dapl_evd_qp_async_error_callb.c | 1 + dapl/common/dapl_evd_un_async_error_callb.c | 1 + dapl/common/dapl_evd_util.c | 15 ++ dapl/common/dapl_ia_util.c | 14 ++ dapl/common/dapl_lmr_free.c | 11 +- dapl/common/dapl_psp_create.c | 3 +- dapl/common/dapl_psp_create_any.c | 2 + dapl/common/dapl_psp_free.c | 2 + dapl/common/dapl_pz_create.c | 1 + dapl/common/dapl_pz_free.c | 2 + dapl/common/dapl_rmr_create.c | 2 +- dapl/common/dapl_rmr_free.c | 3 + dapl/common/dapl_rsp_create.c | 2 + dapl/common/dapl_rsp_free.c | 2 + dapl/common/dapl_srq_create.c | 2 + dapl/common/dapl_srq_free.c | 2 + dapl/common/dapl_timer_util.c | 2 - dapl/ibal/dapl_ibal_extensions.c | 2 - dapl/include/dapl.h | 10 ++ dapl/include/dapl_debug.h | 55 ++------ dapl/openib_cma/dapl_ib_dto.h | 62 ++++++++- dapl/openib_cma/dapl_ib_extensions.c | 35 +++++- dapl/openib_cma/dapl_ib_util.c | 5 + dapl/openib_scm/dapl_ib_dto.h | 74 +++++++++- dapl/openib_scm/dapl_ib_extensions.c | 35 +++++- dapl/openib_scm/dapl_ib_util.c | 5 + dapl/udapl/dapl_evd_create.c | 3 +- dapl/udapl/dapl_evd_wait.c | 6 +- dapl/udapl/dapl_init.c | 2 - dapl/udapl/dapl_lmr_create.c | 2 + dapl/udapl/linux/dapl_osd.h | 9 +- dat/include/dat2/dat_ib_extensions.h | 133 +++++++++++++++++- test/dtest/dtestx.c | 88 +++++++++++- 49 files changed, 706 insertions(+), 138 deletions(-) diff --git a/dapl/common/dapl_debug.c b/dapl/common/dapl_debug.c index e717591..7601fc8 100644 --- a/dapl/common/dapl_debug.c +++ b/dapl/common/dapl_debug.c @@ -25,7 +25,6 @@ * and/or other materials provided with the distribution. */ -#include "dapl_debug.h" #include "dapl.h" #if !defined(__KDAPL__) #include @@ -67,52 +66,177 @@ void dapl_internal_dbg_log ( DAPL_DBG_TYPE type, const char *fmt, ...) } } -#if defined(DAPL_COUNTERS) -int dapl_dbg_counters[DCNT_NUM_COUNTERS] = { 0 }; +#ifdef DAPL_COUNTERS /* - * The order of this list must match exactly with the #defines - * in dapl_debug.h + * The order of this list must match the DAT counter definitions */ -char *dapl_dbg_counter_names[] = { - "dapl_ep_create", - "dapl_ep_free", - "dapl_ep_connect", - "dapl_ep_disconnect", - "dapl_ep_post_send", - "dapl_ep_post_recv", - "dapl_ep_post_rdma_write", - "dapl_ep_post_rdma_read", - "dapl_evd_create", - "dapl_evd_free", - "dapl_evd_wait", - "dapl_evd_blocked", - "dapl_evd_completion_notify", - "dapl_evd_dto_callback", - "dapl_evd_connection_callback", - "dapl_evd_dequeue", - "dapl_evd_poll", - "dapl_evd_found", - "dapl_evd_not_found", - "dapls_timer_set", - "dapls_timer_cancel", - "dapl_extensions" +static char *ia_cntr_names[] = { + "DCNT_IA_PZ_CREATE", + "DCNT_IA_PZ_FREE", + "DCNT_IA_LMR_CREATE", + "DCNT_IA_LMR_FREE", + "DCNT_IA_RMR_CREATE", + "DCNT_IA_RMR_FREE", + "DCNT_IA_PSP_CREATE", + "DCNT_IA_PSP_CREATE_ANY", + "DCNT_IA_PSP_FREE", + "DCNT_IA_RSP_CREATE", + "DCNT_IA_RSP_FREE", + "DCNT_IA_EVD_CREATE", + "DCNT_IA_EVD_FREE", + "DCNT_IA_EP_CREATE", + "DCNT_IA_EP_FREE", + "DCNT_IA_SRQ_CREATE", + "DCNT_IA_SRQ_FREE", + "DCNT_IA_SP_CR", + "DCNT_IA_SP_CR_ACCEPTED", + "DCNT_IA_SP_CR_REJECTED", + "DCNT_IA_MEM_ALLOC", + "DCNT_IA_MEM_ALLOC_DATA", + "DCNT_IA_MEM_FREE", + "DCNT_IA_ASYNC_ERROR", + "DCNT_IA_ASYNC_QP_ERROR", + "DCNT_IA_ASYNC_CQ_ERROR" }; -void dapl_dump_cntr( int cntr ) +static char *ep_cntr_names[] = { + "DCNT_EP_CONNECT", + "DCNT_EP_DISCONNECT", + "DCNT_EP_POST_SEND", + "DCNT_EP_POST_SEND_DATA", + "DCNT_EP_POST_SEND_UD", + "DCNT_EP_POST_SEND_UD_DATA", + "DCNT_EP_POST_RECV", + "DCNT_EP_POST_RECV_DATA", + "DCNT_EP_POST_WRITE", + "DCNT_EP_POST_WRITE_DATA", + "DCNT_EP_POST_WRITE_IMM", + "DCNT_EP_POST_WRITE_IMM_DATA", + "DCNT_EP_POST_READ", + "DCNT_EP_POST_READ_DATA", + "DCNT_EP_POST_CMP_SWAP", + "DCNT_EP_POST_FETCH_ADD", + "DCNT_EP_RECV", + "DCNT_EP_RECV_DATA", + "DCNT_EP_RECV_UD", + "DCNT_EP_RECV_UD_DATA", + "DCNT_EP_RECV_IMM", + "DCNT_EP_RECV_IMM_DATA", + "DCNT_EP_RECV_RDMA_IMM", + "DCNT_EP_RECV_RDMA_IMM_DATA", +}; + +static char *evd_cntr_names[] = { + "DCNT_EVD_WAIT", + "DCNT_EVD_WAIT_BLOCKED", + "DCNT_EVD_WAIT_NOTIFY", + "DCNT_EVD_DEQUEUE", + "DCNT_EVD_DEQUEUE_FOUND", + "DCNT_EVD_DEQUEUE_NOT_FOUND", + "DCNT_EVD_DEQUEUE_POLL", + "DCNT_EVD_DEQUEUE_POLL_FOUND", + "DCNT_EVD_CONN_CALLBACK", + "DCNT_EVD_DTO_CALLBACK", +}; + +DAT_RETURN dapl_query_counter(DAT_HANDLE dh, + int counter, + void *p_cntrs_out, + int reset) { - int i; + int i, max; + DAT_UINT64 *p_cntrs; + DAT_HANDLE_TYPE type = 0; + + dat_get_handle_type(dh, &type); - for ( i = 0; i < DCNT_NUM_COUNTERS; i++ ) - { - if (( cntr == i ) || ( cntr == DCNT_ALL_COUNTERS )) - { - dapl_dbg_log ( DAPL_DBG_TYPE_CNTR, - "DAPL Counter: %s = %lu \n", - dapl_dbg_counter_names[i], - dapl_dbg_counters[i] ); + switch(type) { + case DAT_HANDLE_TYPE_IA: + max = DCNT_IA_ALL_COUNTERS; + p_cntrs = ((DAPL_IA*)dh)->cntrs; + break; + case DAT_HANDLE_TYPE_EP: + max = DCNT_EP_ALL_COUNTERS; + p_cntrs = ((DAPL_EP*)dh)->cntrs; + break; + case DAT_HANDLE_TYPE_EVD: + max = DCNT_EVD_ALL_COUNTERS; + p_cntrs = ((DAPL_EVD*)dh)->cntrs; + break; + default: + return DAT_INVALID_HANDLE; + } + + for (i=0; i < max; i++) { + if ((counter == i) || (counter == max)) { + ((DAT_UINT64*)p_cntrs_out)[i] = p_cntrs[i]; + if (reset) p_cntrs[i]=0; + } + } + return DAT_SUCCESS; +} + +char *dapl_query_counter_name(DAT_HANDLE dh, int counter) +{ + DAT_HANDLE_TYPE type = 0; + + dat_get_handle_type(dh, &type); + + switch(type) { + case DAT_HANDLE_TYPE_IA: + if (counter < DCNT_IA_ALL_COUNTERS) + return ia_cntr_names[counter]; + break; + case DAT_HANDLE_TYPE_EP: + if (counter < DCNT_EP_ALL_COUNTERS) + return ep_cntr_names[counter]; + break; + case DAT_HANDLE_TYPE_EVD: + if (counter < DCNT_EVD_ALL_COUNTERS) + return evd_cntr_names[counter]; + break; + default: + return NULL; + } + return NULL; +} + +#include +void dapl_print_counter(DAT_HANDLE dh, int counter, int reset) +{ + int i, max; + DAT_UINT64 *p_cntrs; + DAT_HANDLE_TYPE type = 0; + + dat_get_handle_type(dh, &type); + + switch(type) { + case DAT_HANDLE_TYPE_IA: + max = DCNT_IA_ALL_COUNTERS; + p_cntrs = ((DAPL_IA*)dh)->cntrs; + break; + case DAT_HANDLE_TYPE_EP: + max = DCNT_EP_ALL_COUNTERS; + p_cntrs = ((DAPL_EP*)dh)->cntrs; + break; + case DAT_HANDLE_TYPE_EVD: + max = DCNT_EVD_ALL_COUNTERS; + p_cntrs = ((DAPL_EVD*)dh)->cntrs; + break; + default: + return; + } + + for (i=0; i < max; i++) { + if ((counter == i) || (counter == max)) { + printf( " %s = " F64u " \n", + dapl_query_counter_name(dh, i), + p_cntrs[i] ); + if (reset) p_cntrs[i]=0; + } } - } + return; } #endif /* DAPL_COUNTERS */ diff --git a/dapl/common/dapl_ep_connect.c b/dapl/common/dapl_ep_connect.c index 0c3f10a..9a72531 100755 --- a/dapl/common/dapl_ep_connect.c +++ b/dapl/common/dapl_ep_connect.c @@ -97,7 +97,6 @@ dapl_ep_connect ( private_data, qos, connect_flags); - DAPL_CNTR (DCNT_EP_CONNECT); dat_status = DAT_SUCCESS; ep_ptr = (DAPL_EP *) ep_handle; @@ -124,6 +123,7 @@ dapl_ep_connect ( dat_status = DAT_ERROR (DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); goto bail; } + DAPL_CNTR (ep_ptr, DCNT_EP_CONNECT); /* * If the endpoint needs a QP, associated the QP with it. diff --git a/dapl/common/dapl_ep_create.c b/dapl/common/dapl_ep_create.c index ff233b0..8982876 100644 --- a/dapl/common/dapl_ep_create.c +++ b/dapl/common/dapl_ep_create.c @@ -89,7 +89,6 @@ dapl_ep_create ( connect_evd_handle, ep_attr, ep_handle); - DAPL_CNTR(DCNT_EP_CREATE); ia_ptr = (DAPL_IA *)ia_handle; dat_status = DAT_SUCCESS; @@ -102,6 +101,7 @@ dapl_ep_create ( dat_status = DAT_ERROR (DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); goto bail; } + DAPL_CNTR(ia_ptr, DCNT_IA_EP_CREATE); /* * Verify non-required parameters. diff --git a/dapl/common/dapl_ep_disconnect.c b/dapl/common/dapl_ep_disconnect.c index fabce92..631d809 100644 --- a/dapl/common/dapl_ep_disconnect.c +++ b/dapl/common/dapl_ep_disconnect.c @@ -75,7 +75,6 @@ dapl_ep_disconnect ( "dapl_ep_disconnect (%p, %x)\n", ep_handle, disconnect_flags); - DAPL_CNTR(DCNT_EP_DISCONNECT); ep_ptr = (DAPL_EP *) ep_handle; @@ -87,6 +86,7 @@ dapl_ep_disconnect ( dat_status = DAT_ERROR (DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } + DAPL_CNTR(ep_ptr, DCNT_EP_DISCONNECT); /* * Do the verification of parameters and the state change diff --git a/dapl/common/dapl_ep_free.c b/dapl/common/dapl_ep_free.c index 62d9644..76b34f0 100644 --- a/dapl/common/dapl_ep_free.c +++ b/dapl/common/dapl_ep_free.c @@ -73,7 +73,6 @@ dapl_ep_free ( dapl_dbg_log (DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_free (%p)\n", ep_handle); - DAPL_CNTR(DCNT_EP_FREE); ep_ptr = (DAPL_EP *) ep_handle; param = &ep_ptr->param; @@ -86,6 +85,7 @@ dapl_ep_free ( dat_status = DAT_ERROR (DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); goto bail; } + DAPL_CNTR(ep_ptr->header.owner_ia, DCNT_IA_EP_FREE); if ( ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED || ep_ptr->param.ep_state == DAT_EP_STATE_PASSIVE_CONNECTION_PENDING || diff --git a/dapl/common/dapl_ep_get_status.c b/dapl/common/dapl_ep_get_status.c index 3af7f9a..3f856d3 100644 --- a/dapl/common/dapl_ep_get_status.c +++ b/dapl/common/dapl_ep_get_status.c @@ -99,12 +99,12 @@ dapl_ep_get_status ( if ( in_dto_idle != NULL ) { - *in_dto_idle = (dapls_cb_pending(&ep_ptr->recv_buffer)) ? DAT_FALSE : DAT_TRUE; + *in_dto_idle = dapls_cb_pending(&ep_ptr->recv_buffer); } if ( out_dto_idle != NULL ) { - *out_dto_idle = (dapls_cb_pending(&ep_ptr->req_buffer)) ? DAT_FALSE : DAT_TRUE; + *out_dto_idle = dapls_cb_pending(&ep_ptr->req_buffer); } bail: diff --git a/dapl/common/dapl_ep_post_rdma_read.c b/dapl/common/dapl_ep_post_rdma_read.c index 95a7e05..dc6c39c 100644 --- a/dapl/common/dapl_ep_post_rdma_read.c +++ b/dapl/common/dapl_ep_post_rdma_read.c @@ -86,7 +86,6 @@ dapl_ep_post_rdma_read ( user_cookie.as_64, remote_iov, completion_flags); - DAPL_CNTR(DCNT_POST_RDMA_READ); dat_status = dapl_ep_post_send_req(ep_handle, num_segments, diff --git a/dapl/common/dapl_ep_post_rdma_write.c b/dapl/common/dapl_ep_post_rdma_write.c index 6b9ae94..ef79d0c 100644 --- a/dapl/common/dapl_ep_post_rdma_write.c +++ b/dapl/common/dapl_ep_post_rdma_write.c @@ -85,7 +85,6 @@ dapl_ep_post_rdma_write ( user_cookie.as_64, remote_iov, completion_flags); - DAPL_CNTR(DCNT_POST_RDMA_WRITE); dat_status = dapl_ep_post_send_req(ep_handle, num_segments, diff --git a/dapl/common/dapl_ep_post_recv.c b/dapl/common/dapl_ep_post_recv.c index 12724cd..b11eb96 100644 --- a/dapl/common/dapl_ep_post_recv.c +++ b/dapl/common/dapl_ep_post_recv.c @@ -85,7 +85,6 @@ dapl_ep_post_recv ( local_iov, user_cookie.as_64, completion_flags); - DAPL_CNTR (DCNT_POST_RECV); if ( DAPL_BAD_HANDLE (ep_handle, DAPL_MAGIC_EP) ) { diff --git a/dapl/common/dapl_ep_post_send.c b/dapl/common/dapl_ep_post_send.c index c13a095..0229132 100644 --- a/dapl/common/dapl_ep_post_send.c +++ b/dapl/common/dapl_ep_post_send.c @@ -84,7 +84,6 @@ dapl_ep_post_send ( local_iov, user_cookie.as_64, completion_flags); - DAPL_CNTR(DCNT_POST_SEND); dat_status = dapl_ep_post_send_req(ep_handle, num_segments, diff --git a/dapl/common/dapl_ep_util.c b/dapl/common/dapl_ep_util.c index 782e09a..f75c059 100644 --- a/dapl/common/dapl_ep_util.c +++ b/dapl/common/dapl_ep_util.c @@ -125,6 +125,16 @@ dapl_ep_alloc ( /* zero the structure */ dapl_os_memzero (ep_ptr, sizeof (DAPL_EP) + sizeof (DAT_SOCK_ADDR)); +#ifdef DAPL_COUNTERS + /* Allocate counters */ + ep_ptr->cntrs = dapl_os_alloc(sizeof(DAT_UINT64) * DCNT_EP_ALL_COUNTERS); + if (ep_ptr->cntrs == NULL) { + dapl_os_free(ep_ptr, sizeof (DAPL_EP) + sizeof (DAT_SOCK_ADDR)); + return (NULL); + } + dapl_os_memzero (ep_ptr->cntrs, sizeof(DAT_UINT64) * DCNT_EP_ALL_COUNTERS); +#endif /* DAPL_COUNTERS */ + /* * initialize the header */ @@ -227,6 +237,11 @@ dapl_ep_dealloc ( ep_ptr->ibal_cm_handle = NULL; } #endif + +#ifdef DAPL_COUNTERS + dapl_os_free(ep_ptr->cntrs, sizeof(DAT_UINT64) * DCNT_EP_ALL_COUNTERS); +#endif /* DAPL_COUNTERS */ + dapl_os_free (ep_ptr, sizeof (DAPL_EP) + sizeof (DAT_SOCK_ADDR) ); } diff --git a/dapl/common/dapl_evd_connection_callb.c b/dapl/common/dapl_evd_connection_callb.c index 7f994b0..d5a9750 100644 --- a/dapl/common/dapl_evd_connection_callb.c +++ b/dapl/common/dapl_evd_connection_callb.c @@ -84,8 +84,6 @@ dapl_evd_connection_callback ( ib_cm_event, (void *) ib_cm_handle ); - DAPL_CNTR(DCNT_EVD_CONN_CALLBACK); - /* * Determine the type of handle passed back to us in the context * and sort out key parameters. @@ -101,6 +99,7 @@ dapl_evd_connection_callback ( */ ep_ptr = (DAPL_EP *) context; evd_ptr = (DAPL_EVD *) ep_ptr->param.connect_evd_handle; + DAPL_CNTR(evd_ptr, DCNT_EVD_CONN_CALLBACK); prd_ptr = (DAPL_PRIVATE *)private_data_ptr; private_data_size = 0; diff --git a/dapl/common/dapl_evd_cq_async_error_callb.c b/dapl/common/dapl_evd_cq_async_error_callb.c index 2b0251a..ca4bc8d 100644 --- a/dapl/common/dapl_evd_cq_async_error_callb.c +++ b/dapl/common/dapl_evd_cq_async_error_callb.c @@ -79,6 +79,7 @@ dapl_evd_cq_async_error_callback ( evd = (DAPL_EVD *) context; async_evd = evd->header.owner_ia->async_error_evd; + DAPL_CNTR(evd->header.owner_ia, DCNT_IA_ASYNC_CQ_ERROR); dat_status = dapls_evd_post_async_error_event( async_evd, diff --git a/dapl/common/dapl_evd_dequeue.c b/dapl/common/dapl_evd_dequeue.c index f6faeeb..5a409f3 100644 --- a/dapl/common/dapl_evd_dequeue.c +++ b/dapl/common/dapl_evd_dequeue.c @@ -75,7 +75,6 @@ DAT_RETURN DAT_API dapl_evd_dequeue ( "dapl_evd_dequeue (%p, %p)\n", evd_handle, event); - DAPL_CNTR(DCNT_EVD_DEQUEUE); evd_ptr = (DAPL_EVD *)evd_handle; dat_status = DAT_SUCCESS; @@ -91,6 +90,7 @@ DAT_RETURN DAT_API dapl_evd_dequeue ( dat_status = DAT_ERROR (DAT_INVALID_PARAMETER,DAT_INVALID_ARG2); goto bail; } + DAPL_CNTR(evd_ptr, DCNT_EVD_DEQUEUE); /* * We need to dequeue under lock, as the IB OS Access API @@ -122,17 +122,18 @@ DAT_RETURN DAT_API dapl_evd_dequeue ( *event = *local_event; dat_status = dapls_rbuf_add (&evd_ptr->free_event_queue, local_event); - DAPL_CNTR(DCNT_EVD_DEQUEUE_FOUND); + DAPL_CNTR(evd_ptr, DCNT_EVD_DEQUEUE_FOUND); + } else if (evd_ptr->ib_cq_handle != IB_INVALID_HANDLE) { dat_status = dapls_evd_cq_poll_to_event(evd_ptr, event); - DAPL_CNTR(DCNT_EVD_DEQUEUE_POLL); + DAPL_CNTR(evd_ptr, DCNT_EVD_DEQUEUE_POLL); } else { dat_status = DAT_ERROR (DAT_QUEUE_EMPTY,0); - DAPL_CNTR(DCNT_EVD_DEQUEUE_NOT_FOUND); + DAPL_CNTR(evd_ptr, DCNT_EVD_DEQUEUE_NOT_FOUND); } dapl_os_unlock ( &evd_ptr->header.lock ); diff --git a/dapl/common/dapl_evd_dto_callb.c b/dapl/common/dapl_evd_dto_callb.c index bb3e1f8..347c2e9 100755 --- a/dapl/common/dapl_evd_dto_callb.c +++ b/dapl/common/dapl_evd_dto_callb.c @@ -84,9 +84,9 @@ dapl_evd_dto_callback ( hca_handle, cq_handle, user_context); - DAPL_CNTR(DCNT_EVD_DTO_CALLBACK); evd_ptr = (DAPL_EVD *) user_context; + DAPL_CNTR(evd_ptr, DCNT_EVD_DTO_CALLBACK); dapl_os_assert (hca_handle == evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle); dapl_os_assert (evd_ptr->ib_cq_handle == cq_handle); diff --git a/dapl/common/dapl_evd_free.c b/dapl/common/dapl_evd_free.c index 4d126b2..8e4d082 100755 --- a/dapl/common/dapl_evd_free.c +++ b/dapl/common/dapl_evd_free.c @@ -68,7 +68,6 @@ DAT_RETURN DAT_API dapl_evd_free ( DAT_RETURN dat_status; dapl_dbg_log (DAPL_DBG_TYPE_API, "dapl_evd_free (%p)\n", evd_handle); - DAPL_CNTR (DCNT_EVD_FREE); dat_status = DAT_SUCCESS; evd_ptr = (DAPL_EVD *)evd_handle; @@ -79,6 +78,8 @@ DAT_RETURN DAT_API dapl_evd_free ( goto bail; } + DAPL_CNTR(evd_ptr->header.owner_ia, DCNT_IA_EVD_FREE); + if (dapl_os_atomic_read (&evd_ptr->evd_ref_count) != 0) { dat_status = DAT_ERROR (DAT_INVALID_STATE, DAT_INVALID_STATE_EVD_IN_USE); diff --git a/dapl/common/dapl_evd_qp_async_error_callb.c b/dapl/common/dapl_evd_qp_async_error_callb.c index b14db3c..27b6ddc 100644 --- a/dapl/common/dapl_evd_qp_async_error_callb.c +++ b/dapl/common/dapl_evd_qp_async_error_callb.c @@ -114,6 +114,7 @@ dapl_evd_qp_async_error_callback ( ia_ptr = ep_ptr->header.owner_ia; async_evd = (DAPL_EVD *) ia_ptr->async_error_evd; + DAPL_CNTR(ia_ptr, DCNT_IA_ASYNC_QP_ERROR); dapl_dbg_log ( DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, diff --git a/dapl/common/dapl_evd_un_async_error_callb.c b/dapl/common/dapl_evd_un_async_error_callb.c index 5ba0f3e..987aee2 100644 --- a/dapl/common/dapl_evd_un_async_error_callb.c +++ b/dapl/common/dapl_evd_un_async_error_callb.c @@ -78,6 +78,7 @@ dapl_evd_un_async_error_callback ( } async_evd = (DAPL_EVD *) context; + DAPL_CNTR(async_evd->header.owner_ia, DCNT_IA_ASYNC_ERROR); dat_status = dapls_ib_get_async_event(cause_ptr, &async_event); diff --git a/dapl/common/dapl_evd_util.c b/dapl/common/dapl_evd_util.c index c6c7463..ad3d13d 100644 --- a/dapl/common/dapl_evd_util.c +++ b/dapl/common/dapl_evd_util.c @@ -273,6 +273,17 @@ dapls_evd_alloc ( /* zero the structure */ dapl_os_memzero (evd_ptr, sizeof (DAPL_EVD)); +#ifdef DAPL_COUNTERS + /* Allocate counters */ + evd_ptr->cntrs = dapl_os_alloc(sizeof(DAT_UINT64) * DCNT_EVD_ALL_COUNTERS); + if (evd_ptr->cntrs == NULL) { + dapl_os_free(evd_ptr, sizeof(DAPL_EVD)); + return (NULL); + } + dapl_os_memzero(evd_ptr->cntrs, + sizeof(DAT_UINT64) * DCNT_EVD_ALL_COUNTERS); +#endif /* DAPL_COUNTERS */ + /* * initialize the header */ @@ -542,6 +553,10 @@ dapls_evd_dealloc ( } #endif +#ifdef DAPL_COUNTERS + dapl_os_free(evd_ptr->cntrs, sizeof(DAT_UINT64) * DCNT_EVD_ALL_COUNTERS); +#endif /* DAPL_COUNTERS */ + dapl_os_free (evd_ptr, sizeof (DAPL_EVD)); bail: diff --git a/dapl/common/dapl_ia_util.c b/dapl/common/dapl_ia_util.c index 7b03afc..a42063b 100755 --- a/dapl/common/dapl_ia_util.c +++ b/dapl/common/dapl_ia_util.c @@ -78,6 +78,16 @@ dapl_ia_alloc ( DAT_PROVIDER * provider, DAPL_HCA * hca_ptr ) /* zero the structure */ dapl_os_memzero (ia_ptr, sizeof (DAPL_IA)); +#ifdef DAPL_COUNTERS + /* Allocate counters */ + ia_ptr->cntrs = dapl_os_alloc(sizeof(DAT_UINT64) * DCNT_IA_ALL_COUNTERS); + if (ia_ptr->cntrs == NULL) { + dapl_os_free (ia_ptr, sizeof(DAPL_IA)); + return (NULL); + } + dapl_os_memzero(ia_ptr->cntrs, sizeof(DAT_UINT64) * DCNT_IA_ALL_COUNTERS); +#endif /* DAPL_COUNTERS */ + /* * initialize the header */ @@ -596,6 +606,10 @@ dapls_ia_free ( DAPL_IA *ia_ptr ) ia_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ dapl_os_lock_destroy (&ia_ptr->header.lock); +#ifdef DAPL_COUNTERS + dapl_os_free(ia_ptr->cntrs, sizeof(DAT_UINT64) * DCNT_IA_ALL_COUNTERS); +#endif /* DAPL_COUNTERS */ + dapl_os_free (ia_ptr, sizeof (DAPL_IA)); } diff --git a/dapl/common/dapl_lmr_free.c b/dapl/common/dapl_lmr_free.c index f4106b4..3fa67b2 100644 --- a/dapl/common/dapl_lmr_free.c +++ b/dapl/common/dapl_lmr_free.c @@ -62,6 +62,7 @@ dapl_lmr_free ( IN DAT_LMR_HANDLE lmr_handle ) { DAPL_LMR *lmr; + DAPL_PZ *pz; DAT_RETURN dat_status; dapl_dbg_log (DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr_handle); @@ -72,7 +73,10 @@ dapl_lmr_free ( goto bail; } - lmr = (DAPL_LMR *) lmr_handle; + lmr = (DAPL_LMR *)lmr_handle; + pz = (DAPL_PZ *)lmr->param.pz_handle; + + DAPL_CNTR(pz->header.owner_ia, DCNT_IA_LMR_FREE); switch (lmr->param.mem_type) { @@ -85,8 +89,6 @@ dapl_lmr_free ( case DAT_MEM_TYPE_VIRTUAL: case DAT_MEM_TYPE_LMR: { - DAPL_PZ *pz; - if ( 0 != dapl_os_atomic_read (&lmr->lmr_ref_count) ) { return DAT_INVALID_STATE; @@ -103,10 +105,7 @@ dapl_lmr_free ( if (dat_status == DAT_SUCCESS) { - - pz = (DAPL_PZ *) lmr->param.pz_handle; dapl_os_atomic_dec (&pz->pz_ref_count); - dapl_lmr_dealloc (lmr); } else diff --git a/dapl/common/dapl_psp_create.c b/dapl/common/dapl_psp_create.c index 9d2e945..a6eb31c 100644 --- a/dapl/common/dapl_psp_create.c +++ b/dapl/common/dapl_psp_create.c @@ -114,7 +114,8 @@ dapl_psp_create ( dat_status = DAT_ERROR (DAT_INVALID_PARAMETER,DAT_INVALID_ARG4); goto bail; } - + + DAPL_CNTR(ia_ptr, DCNT_IA_PSP_CREATE); /* * See if we have a quiescent listener to use for this PSP, else diff --git a/dapl/common/dapl_psp_create_any.c b/dapl/common/dapl_psp_create_any.c index e2faa4a..9b9b575 100644 --- a/dapl/common/dapl_psp_create_any.c +++ b/dapl/common/dapl_psp_create_any.c @@ -133,6 +133,8 @@ dapl_psp_create_any ( goto bail; } + DAPL_CNTR(ia_ptr, DCNT_IA_PSP_CREATE_ANY); + /* * Fill out the args for a PSP */ diff --git a/dapl/common/dapl_psp_free.c b/dapl/common/dapl_psp_free.c index 28913b4..0117e19 100644 --- a/dapl/common/dapl_psp_free.c +++ b/dapl/common/dapl_psp_free.c @@ -82,6 +82,8 @@ dapl_psp_free ( ia_ptr = sp_ptr->header.owner_ia; + DAPL_CNTR(ia_ptr->header.owner_ia, DCNT_IA_PSP_FREE); + /* * Remove the connection listener if it has been established * and there are no current connections in progress. diff --git a/dapl/common/dapl_pz_create.c b/dapl/common/dapl_pz_create.c index a7105aa..efca311 100644 --- a/dapl/common/dapl_pz_create.c +++ b/dapl/common/dapl_pz_create.c @@ -90,6 +90,7 @@ dapl_pz_create ( dat_status = DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY); goto bail; } + DAPL_CNTR(ia, DCNT_IA_PZ_CREATE); dat_status = dapls_ib_pd_alloc (ia, pz); if ( DAT_SUCCESS != dat_status ) diff --git a/dapl/common/dapl_pz_free.c b/dapl/common/dapl_pz_free.c index b2510f7..270dd38 100644 --- a/dapl/common/dapl_pz_free.c +++ b/dapl/common/dapl_pz_free.c @@ -72,6 +72,8 @@ dapl_pz_free ( pz = (DAPL_PZ *) pz_handle; + DAPL_CNTR(pz->header.owner_ia, DCNT_IA_PZ_FREE); + if ( 0 != dapl_os_atomic_read (&pz->pz_ref_count) ) { dat_status = DAT_ERROR (DAT_INVALID_STATE,DAT_INVALID_STATE_PZ_IN_USE); diff --git a/dapl/common/dapl_rmr_create.c b/dapl/common/dapl_rmr_create.c index 9a7dbd9..d3e1661 100755 --- a/dapl/common/dapl_rmr_create.c +++ b/dapl/common/dapl_rmr_create.c @@ -71,7 +71,6 @@ dapl_rmr_create ( } pz = (DAPL_PZ *) pz_handle; - rmr = dapl_rmr_alloc (pz); if ( rmr == NULL ) @@ -79,6 +78,7 @@ dapl_rmr_create ( dat_status = DAT_ERROR (DAT_INSUFFICIENT_RESOURCES,DAT_RESOURCE_MEMORY); goto bail; } + DAPL_CNTR(pz->header.owner_ia, DCNT_IA_RMR_CREATE); dat_status = dapls_ib_mw_alloc (rmr); diff --git a/dapl/common/dapl_rmr_free.c b/dapl/common/dapl_rmr_free.c index a4edb13..f75aaed 100644 --- a/dapl/common/dapl_rmr_free.c +++ b/dapl/common/dapl_rmr_free.c @@ -58,6 +58,7 @@ dapl_rmr_free ( IN DAT_RMR_HANDLE rmr_handle ) { DAPL_RMR *rmr; + DAPL_PZ *pz; DAT_RETURN dat_status; dat_status = DAT_SUCCESS; @@ -69,6 +70,8 @@ dapl_rmr_free ( } rmr = (DAPL_RMR *) rmr_handle; + pz = (DAPL_PZ *)rmr->param.pz_handle; + DAPL_CNTR(pz->header.owner_ia, DCNT_IA_RMR_FREE); /* * If the user did not perform an unbind op, release diff --git a/dapl/common/dapl_rsp_create.c b/dapl/common/dapl_rsp_create.c index a9c9b5f..cf41dd9 100644 --- a/dapl/common/dapl_rsp_create.c +++ b/dapl/common/dapl_rsp_create.c @@ -126,6 +126,8 @@ dapl_rsp_create ( goto bail; } + DAPL_CNTR(ia_ptr, DCNT_IA_RSP_CREATE); + sp_ptr = dapls_ia_sp_search (ia_ptr, conn_qual, DAT_FALSE); sp_found = DAT_TRUE; if (sp_ptr == NULL) diff --git a/dapl/common/dapl_rsp_free.c b/dapl/common/dapl_rsp_free.c index b7bc40c..e016f0d 100644 --- a/dapl/common/dapl_rsp_free.c +++ b/dapl/common/dapl_rsp_free.c @@ -85,6 +85,8 @@ dapl_rsp_free ( /* ia_ptr = (DAPL_IA *)sp_ptr->header.owner_ia; */ ia_ptr = sp_ptr->header.owner_ia; + DAPL_CNTR(ia_ptr, DCNT_IA_RSP_FREE); + /* * Remove the connection listener if there are no connections. If * we defer removing the sp it becomes something of a zombie diff --git a/dapl/common/dapl_srq_create.c b/dapl/common/dapl_srq_create.c index b03bbd6..ef45826 100644 --- a/dapl/common/dapl_srq_create.c +++ b/dapl/common/dapl_srq_create.c @@ -97,6 +97,8 @@ dapl_srq_create ( goto bail; } + DAPL_CNTR(ia_ptr, DCNT_IA_SRQ_CREATE); + /* * Verify non-required parameters. * N.B. Assumption: any parameter that can be diff --git a/dapl/common/dapl_srq_free.c b/dapl/common/dapl_srq_free.c index 55d74fc..9decf54 100644 --- a/dapl/common/dapl_srq_free.c +++ b/dapl/common/dapl_srq_free.c @@ -96,6 +96,8 @@ dapl_srq_free ( ia_ptr = srq_ptr->header.owner_ia; + DAPL_CNTR(ia_ptr, DCNT_IA_SRQ_FREE); + /* * Do verification of parameters and the state change atomically. */ diff --git a/dapl/common/dapl_timer_util.c b/dapl/common/dapl_timer_util.c index c7407c1..12c41bc 100644 --- a/dapl/common/dapl_timer_util.c +++ b/dapl/common/dapl_timer_util.c @@ -107,7 +107,6 @@ dapls_timer_set ( DAPL_OS_TIMEVAL cur_time; DAT_BOOLEAN wakeup_tmo_thread; - DAPL_CNTR(DCNT_TIMER_SET); /* * Start the timer thread the first time we need a timer */ @@ -216,7 +215,6 @@ void dapls_timer_cancel ( IN DAPL_OS_TIMER *timer) { - DAPL_CNTR(DCNT_TIMER_CANCEL); dapl_os_lock ( &g_daplTimerHead.lock ); /* * make sure the entry has not been removed by another thread diff --git a/dapl/ibal/dapl_ibal_extensions.c b/dapl/ibal/dapl_ibal_extensions.c index b05c0bb..643125d 100644 --- a/dapl/ibal/dapl_ibal_extensions.c +++ b/dapl/ibal/dapl_ibal_extensions.c @@ -93,8 +93,6 @@ dapl_extensions(IN DAT_HANDLE dat_handle, "dapl_extensions(hdl %p operation %d, ...)\n", dat_handle, ext_op); - DAPL_CNTR(DCNT_EXTENSION); - switch ((int)ext_op) { diff --git a/dapl/include/dapl.h b/dapl/include/dapl.h index e2025ce..b13c963 100755 --- a/dapl/include/dapl.h +++ b/dapl/include/dapl.h @@ -313,6 +313,9 @@ struct dapl_ia DAPL_LLIST_HEAD psp_list_head; /* PSP queue */ DAPL_LLIST_HEAD rsp_list_head; /* RSP queue */ DAPL_LLIST_HEAD srq_list_head; /* SRQ queue */ +#ifdef DAPL_COUNTERS + void *cntrs; +#endif }; /* DAPL_CNO maps to DAT_CNO_HANDLE */ @@ -388,6 +391,10 @@ struct dapl_evd DAT_COUNT threshold; DAPL_EVD_COMPLETION completion_type; + +#ifdef DAPL_COUNTERS + void *cntrs; +#endif }; /* DAPL_PRIVATE used to pass private data in a connection */ @@ -472,6 +479,9 @@ struct dapl_ep DAT_BOOLEAN sent_discreq; dp_ib_cm_handle_t ibal_cm_handle; #endif +#ifdef DAPL_COUNTERS + void *cntrs; +#endif }; /* DAPL_SRQ maps to DAT_SRQ_HANDLE */ diff --git a/dapl/include/dapl_debug.h b/dapl/include/dapl_debug.h index f0de7c8..d034227 100644 --- a/dapl/include/dapl_debug.h +++ b/dapl/include/dapl_debug.h @@ -99,50 +99,25 @@ static __inline void dapl_dbg_log ( DAPL_DBG_TYPE type, const char *fmt, ...) #endif /* !DAPL_DBG */ -/* - * Counters - */ -#define DCNT_EP_CREATE 0 -#define DCNT_EP_FREE 1 -#define DCNT_EP_CONNECT 2 -#define DCNT_EP_DISCONNECT 3 -#define DCNT_POST_SEND 4 -#define DCNT_POST_RECV 5 -#define DCNT_POST_RDMA_WRITE 6 -#define DCNT_POST_RDMA_READ 7 -#define DCNT_EVD_CREATE 8 -#define DCNT_EVD_FREE 9 -#define DCNT_EVD_WAIT 10 -#define DCNT_EVD_WAIT_BLOCKED 11 -#define DCNT_EVD_WAIT_CMP_NTFY 12 -#define DCNT_EVD_DTO_CALLBACK 13 -#define DCNT_EVD_CONN_CALLBACK 14 -#define DCNT_EVD_DEQUEUE 15 -#define DCNT_EVD_DEQUEUE_POLL 16 -#define DCNT_EVD_DEQUEUE_FOUND 17 -#define DCNT_EVD_DEQUEUE_NOT_FOUND 18 -#define DCNT_TIMER_SET 19 -#define DCNT_TIMER_CANCEL 20 -#define DCNT_EXTENSION 21 -#define DCNT_NUM_COUNTERS 22 -#define DCNT_ALL_COUNTERS DCNT_NUM_COUNTERS - -#if defined(DAPL_COUNTERS) - -extern void dapl_dump_cntr( int cntr ); -extern int dapl_dbg_counters[]; - -#define DAPL_CNTR(cntr) dapl_os_atomic_inc (&dapl_dbg_counters[cntr]); -#define DAPL_DUMP_CNTR(cntr) dapl_dump_cntr( cntr ); -#define DAPL_COUNTERS_INIT() +#include + +#ifdef DAPL_COUNTERS + +#define DAPL_CNTR(h_ptr, cntr) ((DAT_UINT64*)h_ptr->cntrs)[cntr]++ +#define DAPL_CNTR_DATA(h_ptr, cntr, data) ((DAT_UINT64*)h_ptr->cntrs)[cntr]+= data + +DAT_RETURN dapl_query_counter(DAT_HANDLE dh, + int counter, + void *p_cntrs_out, + int reset); +char *dapl_query_counter_name(DAT_HANDLE dh, int counter); +void dapl_print_counter(DAT_HANDLE dh, int counter, int reset); #else -#define DAPL_CNTR(cntr) -#define DAPL_DUMP_CNTR(cntr) -#define DAPL_COUNTERS_INIT() +#define DAPL_CNTR(handle, cntr) +#define DAPL_CNTR_DATA(handle, cntr, data) #endif /* DAPL_COUNTERS */ - #endif /* _DAPL_DEBUG_H_ */ diff --git a/dapl/openib_cma/dapl_ib_dto.h b/dapl/openib_cma/dapl_ib_dto.h index 2b01963..eba19b7 100644 --- a/dapl/openib_cma/dapl_ib_dto.h +++ b/dapl/openib_cma/dapl_ib_dto.h @@ -89,7 +89,7 @@ dapls_ib_post_recv ( if (NULL == ds_array_p) return (DAT_INSUFFICIENT_RESOURCES); - + /* setup work request */ total_len = 0; wr.next = 0; @@ -126,6 +126,9 @@ dapls_ib_post_recv ( if (ret) return( dapl_convert_errno(errno,"ibv_recv") ); + DAPL_CNTR(ep_ptr, DCNT_EP_POST_RECV); + DAPL_CNTR_DATA(ep_ptr, DCNT_EP_POST_RECV_DATA, total_len); + return DAT_SUCCESS; } @@ -237,6 +240,25 @@ dapls_ib_post_send ( if (ret) return( dapl_convert_errno(errno,"ibv_send") ); +#ifdef DAPL_COUNTERS + switch (op_type) { + case OP_SEND: + DAPL_CNTR(ep_ptr, DCNT_EP_POST_SEND); + DAPL_CNTR_DATA(ep_ptr, DCNT_EP_POST_SEND_DATA,total_len); + break; + case OP_RDMA_WRITE: + DAPL_CNTR(ep_ptr, DCNT_EP_POST_WRITE); + DAPL_CNTR_DATA(ep_ptr, DCNT_EP_POST_WRITE_DATA,total_len); + break; + case OP_RDMA_READ: + DAPL_CNTR(ep_ptr, DCNT_EP_POST_READ); + DAPL_CNTR_DATA(ep_ptr, DCNT_EP_POST_READ_DATA,total_len); + break; + default: + break; + } +#endif /* DAPL_COUNTERS */ + dapl_dbg_log(DAPL_DBG_TYPE_EP," post_snd: returned\n"); return DAT_SUCCESS; } @@ -416,7 +438,25 @@ dapls_ib_post_ext_send ( if (ret) return( dapl_convert_errno(errno,"ibv_send") ); - + +#ifdef DAPL_COUNTERS + switch (op_type) { + case OP_RDMA_WRITE_IMM: + DAPL_CNTR(ep_ptr, DCNT_EP_POST_WRITE_IMM); + DAPL_CNTR_DATA(ep_ptr, + DCNT_EP_POST_WRITE_IMM_DATA, total_len); + break; + case OP_COMP_AND_SWAP: + DAPL_CNTR(ep_ptr, DCNT_EP_POST_CMP_SWAP); + break; + case OP_FETCH_AND_ADD: + DAPL_CNTR(ep_ptr, DCNT_EP_POST_FETCH_ADD); + break; + default: + break; + } +#endif /* DAPL_COUNTERS */ + dapl_dbg_log(DAPL_DBG_TYPE_EP," post_snd: returned\n"); return DAT_SUCCESS; } @@ -434,6 +474,10 @@ dapls_ib_optional_prv_dat( /* map Work Completions to DAPL WR operations */ STATIC _INLINE_ int dapls_cqe_opcode(ib_work_completion_t *cqe_p) { +#ifdef DAPL_COUNTERS + DAPL_COOKIE *cookie = (DAPL_COOKIE *)(uintptr_t)cqe_p->wr_id; +#endif /* DAPL_COUNTERS */ + switch (cqe_p->opcode) { case IBV_WC_SEND: return (OP_SEND); @@ -451,11 +495,21 @@ STATIC _INLINE_ int dapls_cqe_opcode(ib_work_completion_t *cqe_p) case IBV_WC_BIND_MW: return (OP_BIND_MW); case IBV_WC_RECV: - if (cqe_p->wc_flags & IBV_WC_WITH_IMM) + if (cqe_p->wc_flags & IBV_WC_WITH_IMM) { + DAPL_CNTR(cookie->ep, DCNT_EP_RECV_IMM); + DAPL_CNTR_DATA(cookie->ep, DCNT_EP_RECV_IMM_DATA, + cqe_p->byte_len); return (OP_RECEIVE_IMM); - else + } else { + DAPL_CNTR(cookie->ep, DCNT_EP_RECV); + DAPL_CNTR_DATA(cookie->ep, DCNT_EP_RECV_DATA, + cqe_p->byte_len); return (OP_RECEIVE); + } case IBV_WC_RECV_RDMA_WITH_IMM: + DAPL_CNTR(cookie->ep, DCNT_EP_RECV_RDMA_IMM); + DAPL_CNTR_DATA(cookie->ep, DCNT_EP_RECV_RDMA_IMM_DATA, + cqe_p->byte_len); return (OP_RECEIVE_IMM); default: return (OP_INVALID); diff --git a/dapl/openib_cma/dapl_ib_extensions.c b/dapl/openib_cma/dapl_ib_extensions.c index 1402057..af34128 100755 --- a/dapl/openib_cma/dapl_ib_extensions.c +++ b/dapl/openib_cma/dapl_ib_extensions.c @@ -94,8 +94,6 @@ dapl_extensions(IN DAT_HANDLE dat_handle, "dapl_extensions(hdl %p operation %d, ...)\n", dat_handle, ext_op); - DAPL_CNTR(DCNT_EXTENSION); - switch ((int)ext_op) { @@ -150,6 +148,39 @@ dapl_extensions(IN DAT_HANDLE dat_handle, break; +#ifdef DAPL_COUNTERS + case DAT_QUERY_COUNTERS_OP: + { + int cntr, reset; + DAT_UINT64 * p_cntr_out; + + dapl_dbg_log(DAPL_DBG_TYPE_RTN, + " Query counter extension call\n"); + + cntr = va_arg(args, int); + p_cntr_out = va_arg(args, DAT_UINT64*); + reset = va_arg(args, int); + + status = dapl_query_counter(dat_handle, cntr, + p_cntr_out, reset); + break; + } + case DAT_PRINT_COUNTERS_OP: + { + int cntr, reset; + + dapl_dbg_log(DAPL_DBG_TYPE_RTN, + " Print counter extension call\n"); + + cntr = va_arg(args, int); + reset = va_arg(args, int); + + dapl_print_counter(dat_handle, cntr, reset); + status = DAT_SUCCESS; + break; + } +#endif /* DAPL_COUNTERS */ + default: dapl_dbg_log(DAPL_DBG_TYPE_ERR, "unsupported extension(%d)\n", (int)ext_op); diff --git a/dapl/openib_cma/dapl_ib_util.c b/dapl/openib_cma/dapl_ib_util.c index 36b534e..c5641f0 100755 --- a/dapl/openib_cma/dapl_ib_util.c +++ b/dapl/openib_cma/dapl_ib_util.c @@ -970,6 +970,11 @@ DAT_NAMED_ATTR ib_attrs[] = { { DAT_IB_ATTR_IMMED_DATA, "TRUE" }, +#ifdef DAPL_COUNTERS + { + DAT_ATTR_COUNTERS, "TRUE" + }, +#endif /* DAPL_COUNTERS */ #endif }; diff --git a/dapl/openib_scm/dapl_ib_dto.h b/dapl/openib_scm/dapl_ib_dto.h index fa19d01..ff338fc 100644 --- a/dapl/openib_scm/dapl_ib_dto.h +++ b/dapl/openib_scm/dapl_ib_dto.h @@ -92,7 +92,7 @@ dapls_ib_post_recv ( if (NULL == ds_array_p) return (DAT_INSUFFICIENT_RESOURCES); - + /* setup work request */ total_len = 0; wr.next = 0; @@ -129,6 +129,9 @@ dapls_ib_post_recv ( if (ret) return( dapl_convert_errno(errno,"ibv_recv") ); + DAPL_CNTR(ep_ptr, DCNT_EP_POST_RECV); + DAPL_CNTR_DATA(ep_ptr, DCNT_EP_POST_RECV_DATA, total_len); + return DAT_SUCCESS; } @@ -244,6 +247,25 @@ dapls_ib_post_send ( if (ret) return( dapl_convert_errno(errno,"ibv_send") ); +#ifdef DAPL_COUNTERS + switch (op_type) { + case OP_SEND: + DAPL_CNTR(ep_ptr, DCNT_EP_POST_SEND); + DAPL_CNTR_DATA(ep_ptr, DCNT_EP_POST_SEND_DATA,total_len); + break; + case OP_RDMA_WRITE: + DAPL_CNTR(ep_ptr, DCNT_EP_POST_WRITE); + DAPL_CNTR_DATA(ep_ptr, DCNT_EP_POST_WRITE_DATA,total_len); + break; + case OP_RDMA_READ: + DAPL_CNTR(ep_ptr, DCNT_EP_POST_READ); + DAPL_CNTR_DATA(ep_ptr, DCNT_EP_POST_READ_DATA,total_len); + break; + default: + break; + } +#endif /* DAPL_COUNTERS */ + dapl_dbg_log(DAPL_DBG_TYPE_EP," post_snd: returned\n"); return DAT_SUCCESS; } @@ -452,8 +474,30 @@ dapls_ib_post_ext_send ( if (ret) return( dapl_convert_errno(errno,"ibv_send") ); +#ifdef DAPL_COUNTERS + switch (op_type) { + case OP_RDMA_WRITE_IMM: + DAPL_CNTR(ep_ptr, DCNT_EP_POST_WRITE_IMM); + DAPL_CNTR_DATA(ep_ptr, + DCNT_EP_POST_WRITE_IMM_DATA, total_len); + break; + case OP_COMP_AND_SWAP: + DAPL_CNTR(ep_ptr, DCNT_EP_POST_CMP_SWAP); + break; + case OP_FETCH_AND_ADD: + DAPL_CNTR(ep_ptr, DCNT_EP_POST_FETCH_ADD); + break; + case OP_SEND_UD: + DAPL_CNTR(ep_ptr, DCNT_EP_POST_SEND_UD); + DAPL_CNTR_DATA(ep_ptr, DCNT_EP_POST_SEND_UD_DATA, total_len); + break; + default: + break; + } +#endif /* DAPL_COUNTERS */ + dapl_dbg_log(DAPL_DBG_TYPE_EP," post_snd: returned\n"); - return DAT_SUCCESS; + return DAT_SUCCESS; } #endif @@ -470,6 +514,10 @@ dapls_ib_optional_prv_dat( /* map Work Completions to DAPL WR operations */ STATIC _INLINE_ int dapls_cqe_opcode(ib_work_completion_t *cqe_p) { +#ifdef DAPL_COUNTERS + DAPL_COOKIE *cookie = (DAPL_COOKIE *)(uintptr_t)cqe_p->wr_id; +#endif /* DAPL_COUNTERS */ + switch (cqe_p->opcode) { case IBV_WC_SEND: if (CQE_WR_TYPE_UD(cqe_p->wr_id)) @@ -490,13 +538,27 @@ STATIC _INLINE_ int dapls_cqe_opcode(ib_work_completion_t *cqe_p) case IBV_WC_BIND_MW: return (OP_BIND_MW); case IBV_WC_RECV: - if (CQE_WR_TYPE_UD(cqe_p->wr_id)) + if (CQE_WR_TYPE_UD(cqe_p->wr_id)) { + DAPL_CNTR(cookie->ep, DCNT_EP_RECV_UD); + DAPL_CNTR_DATA(cookie->ep, DCNT_EP_RECV_UD_DATA, + cqe_p->byte_len); return (OP_RECV_UD); - else if (cqe_p->wc_flags & IBV_WC_WITH_IMM) - return (OP_RECEIVE_MSG_IMM); - else + } + else if (cqe_p->wc_flags & IBV_WC_WITH_IMM) { + DAPL_CNTR(cookie->ep, DCNT_EP_RECV_IMM); + DAPL_CNTR_DATA(cookie->ep, DCNT_EP_RECV_IMM_DATA, + cqe_p->byte_len); + return (OP_RECEIVE_IMM); + } else { + DAPL_CNTR(cookie->ep, DCNT_EP_RECV); + DAPL_CNTR_DATA(cookie->ep, DCNT_EP_RECV_DATA, + cqe_p->byte_len); return (OP_RECEIVE); + } case IBV_WC_RECV_RDMA_WITH_IMM: + DAPL_CNTR(cookie->ep, DCNT_EP_RECV_RDMA_IMM); + DAPL_CNTR_DATA(cookie->ep, DCNT_EP_RECV_RDMA_IMM_DATA, + cqe_p->byte_len); return (OP_RECEIVE_IMM); default: return (OP_INVALID); diff --git a/dapl/openib_scm/dapl_ib_extensions.c b/dapl/openib_scm/dapl_ib_extensions.c index 692c8a8..1d23ca3 100755 --- a/dapl/openib_scm/dapl_ib_extensions.c +++ b/dapl/openib_scm/dapl_ib_extensions.c @@ -96,8 +96,6 @@ dapl_extensions(IN DAT_HANDLE dat_handle, "dapl_extensions(hdl %p operation %d, ...)\n", dat_handle, ext_op); - DAPL_CNTR(DCNT_EXTENSION); - switch ((int)ext_op) { @@ -167,6 +165,39 @@ dapl_extensions(IN DAT_HANDLE dat_handle, OP_SEND_UD, comp_flags, ah); break; +#ifdef DAPL_COUNTERS + case DAT_QUERY_COUNTERS_OP: + { + int cntr, reset; + DAT_UINT64 * p_cntr_out; + + dapl_dbg_log(DAPL_DBG_TYPE_RTN, + " Query counter extension call\n"); + + cntr = va_arg(args, int); + p_cntr_out = va_arg(args, DAT_UINT64*); + reset = va_arg(args, int); + + status = dapl_query_counter(dat_handle, cntr, + p_cntr_out, reset); + break; + } + case DAT_PRINT_COUNTERS_OP: + { + int cntr, reset; + + dapl_dbg_log(DAPL_DBG_TYPE_RTN, + " Print counter extension call\n"); + + cntr = va_arg(args, int); + reset = va_arg(args, int); + + dapl_print_counter(dat_handle, cntr, reset); + status = DAT_SUCCESS; + break; + } +#endif /* DAPL_COUNTERS */ + default: dapl_dbg_log(DAPL_DBG_TYPE_ERR, "unsupported extension(%d)\n", (int)ext_op); diff --git a/dapl/openib_scm/dapl_ib_util.c b/dapl/openib_scm/dapl_ib_util.c index d82d3f5..7623529 100644 --- a/dapl/openib_scm/dapl_ib_util.c +++ b/dapl/openib_scm/dapl_ib_util.c @@ -618,6 +618,11 @@ DAT_NAMED_ATTR ib_attrs[] = { { DAT_IB_ATTR_UD, "TRUE" }, +#ifdef DAPL_COUNTERS + { + DAT_ATTR_COUNTERS, "TRUE" + }, +#endif /* DAPL_COUNTERS */ #endif }; diff --git a/dapl/udapl/dapl_evd_create.c b/dapl/udapl/dapl_evd_create.c index 7687a81..5ca5c22 100644 --- a/dapl/udapl/dapl_evd_create.c +++ b/dapl/udapl/dapl_evd_create.c @@ -95,7 +95,6 @@ DAT_RETURN DAT_API dapl_evd_create ( cno_handle, evd_flags, evd_handle); - DAPL_CNTR(DCNT_EVD_CREATE); ia_ptr = (DAPL_IA *)ia_handle; cno_ptr = (DAPL_CNO *)cno_handle; @@ -109,6 +108,8 @@ DAT_RETURN DAT_API dapl_evd_create ( goto bail; } + DAPL_CNTR(ia_ptr, DCNT_IA_EVD_CREATE); + if ( evd_min_qlen <= 0 ) { dat_status = DAT_ERROR (DAT_INVALID_PARAMETER,DAT_INVALID_ARG2); diff --git a/dapl/udapl/dapl_evd_wait.c b/dapl/udapl/dapl_evd_wait.c index 578041a..9fc0ba2 100644 --- a/dapl/udapl/dapl_evd_wait.c +++ b/dapl/udapl/dapl_evd_wait.c @@ -85,7 +85,6 @@ DAT_RETURN DAT_API dapl_evd_wait ( threshold, event, nmore); - DAPL_CNTR(DCNT_EVD_WAIT); evd_ptr = (DAPL_EVD *)evd_handle; dat_status = DAT_SUCCESS; @@ -122,6 +121,7 @@ DAT_RETURN DAT_API dapl_evd_wait ( dat_status = DAT_ERROR (DAT_INVALID_STATE,0); goto bail; } + DAPL_CNTR(evd_ptr, DCNT_EVD_WAIT); dapl_dbg_log (DAPL_DBG_TYPE_EVD, "dapl_evd_wait: EVD %p, CQ %p\n", @@ -206,7 +206,7 @@ DAT_RETURN DAT_API dapl_evd_wait ( (evd_ptr->completion_type == DAPL_EVD_STATE_SOLICITED_WAIT) ? IB_NOTIFY_ON_SOLIC_COMP : IB_NOTIFY_ON_NEXT_COMP ); - DAPL_CNTR(DCNT_EVD_WAIT_CMP_NTFY); + DAPL_CNTR(evd_ptr, DCNT_EVD_WAIT_NOTIFY); /* FIXME report error */ dapl_os_assert(dat_status == DAT_SUCCESS); @@ -225,7 +225,7 @@ DAT_RETURN DAT_API dapl_evd_wait ( */ evd_ptr->threshold = threshold; - DAPL_CNTR(DCNT_EVD_WAIT_BLOCKED); + DAPL_CNTR(evd_ptr, DCNT_EVD_WAIT_BLOCKED); #ifdef CQ_WAIT_OBJECT if (evd_ptr->cq_wait_obj_handle) diff --git a/dapl/udapl/dapl_init.c b/dapl/udapl/dapl_init.c index a4afba5..f19468b 100644 --- a/dapl/udapl/dapl_init.c +++ b/dapl/udapl/dapl_init.c @@ -115,8 +115,6 @@ void dapl_init ( void ) goto bail; } - DAPL_COUNTERS_INIT(); - return; bail: diff --git a/dapl/udapl/dapl_lmr_create.c b/dapl/udapl/dapl_lmr_create.c index 99b184a..dd93445 100644 --- a/dapl/udapl/dapl_lmr_create.c +++ b/dapl/udapl/dapl_lmr_create.c @@ -489,6 +489,8 @@ dapl_lmr_create ( ia = (DAPL_IA *) ia_handle; pz = (DAPL_PZ *) pz_handle; + DAPL_CNTR(ia, DCNT_IA_LMR_CREATE); + switch (mem_type) { case DAT_MEM_TYPE_VIRTUAL: diff --git a/dapl/udapl/linux/dapl_osd.h b/dapl/udapl/linux/dapl_osd.h index ae02944..974a0f0 100644 --- a/dapl/udapl/linux/dapl_osd.h +++ b/dapl/udapl/linux/dapl_osd.h @@ -539,10 +539,11 @@ DAT_RETURN dapl_os_get_time (DAPL_OS_TIMEVAL *); * ability to define 64 bit formats, which unfortunatly are non standard * in the C compiler world. E.g. %llx for gcc, %I64x for Windows */ -#define F64d "%lld" -#define F64u "%llu" -#define F64x "%llx" -#define F64X "%llX" +#include +#define F64d "%"PRId64 +#define F64u "%"PRIu64 +#define F64x "%"PRIx64 +#define F64X "%"PRIX64 /* diff --git a/dat/include/dat2/dat_ib_extensions.h b/dat/include/dat2/dat_ib_extensions.h index eb10714..2c082b1 100755 --- a/dat/include/dat2/dat_ib_extensions.h +++ b/dat/include/dat2/dat_ib_extensions.h @@ -58,7 +58,21 @@ * DAT_NAMED_ATTR name == extended operations and version, * version_value = version number of extension API */ -#define DAT_IB_EXTENSION_VERSION 202 /* 2.0.2 */ + +/* 2.0.1 - Initial IB extension support, atomic and immed data + * dat_ib_post_fetch_and_add() + * dat_ib_post_cmp_and_swap() + * dat_ib_post_rdma_write_immed() + * + * 2.0.2 - Add UD support, post send and remote_ah via connect events + * dat_ib_post_send_ud() + * + * 2.0.3 - Add query/print counter support for IA, EP, and EVD's + * dat_query_counters(), dat_print_counters() + * + */ +#define DAT_IB_EXTENSION_VERSION 203 /* 2.0.3 */ +#define DAT_ATTR_COUNTERS "DAT_COUNTERS" #define DAT_IB_ATTR_FETCH_AND_ADD "DAT_IB_FETCH_AND_ADD" #define DAT_IB_ATTR_CMP_AND_SWAP "DAT_IB_CMP_AND_SWAP" #define DAT_IB_ATTR_IMMED_DATA "DAT_IB_IMMED_DATA" @@ -87,7 +101,9 @@ typedef enum dat_ib_op DAT_IB_FETCH_AND_ADD_OP, DAT_IB_CMP_AND_SWAP_OP, DAT_IB_RDMA_WRITE_IMMED_OP, - DAT_IB_UD_SEND_OP + DAT_IB_UD_SEND_OP, + DAT_QUERY_COUNTERS_OP, + DAT_PRINT_COUNTERS_OP } DAT_IB_OP; @@ -247,6 +263,92 @@ typedef enum dat_ib_service_type } DAT_IB_SERVICE_TYPE; +/* + * Definitions for 64-bit IA Counters + */ +typedef enum dat_ia_counters +{ + DCNT_IA_PZ_CREATE, + DCNT_IA_PZ_FREE, + DCNT_IA_LMR_CREATE, + DCNT_IA_LMR_FREE, + DCNT_IA_RMR_CREATE, + DCNT_IA_RMR_FREE, + DCNT_IA_PSP_CREATE, + DCNT_IA_PSP_CREATE_ANY, + DCNT_IA_PSP_FREE, + DCNT_IA_RSP_CREATE, + DCNT_IA_RSP_FREE, + DCNT_IA_EVD_CREATE, + DCNT_IA_EVD_FREE, + DCNT_IA_EP_CREATE, + DCNT_IA_EP_FREE, + DCNT_IA_SRQ_CREATE, + DCNT_IA_SRQ_FREE, + DCNT_IA_SP_CR, + DCNT_IA_SP_CR_ACCEPTED, + DCNT_IA_SP_CR_REJECTED, + DCNT_IA_MEM_ALLOC, + DCNT_IA_MEM_ALLOC_DATA, + DCNT_IA_MEM_FREE, + DCNT_IA_ASYNC_ERROR, + DCNT_IA_ASYNC_QP_ERROR, + DCNT_IA_ASYNC_CQ_ERROR, + DCNT_IA_ALL_COUNTERS, /* MUST be last */ + +} DAT_IA_COUNTERS; + +/* + * Definitions for 64-bit EP Counters + */ +typedef enum dat_ep_counters +{ + DCNT_EP_CONNECT, + DCNT_EP_DISCONNECT, + DCNT_EP_POST_SEND, + DCNT_EP_POST_SEND_DATA, + DCNT_EP_POST_SEND_UD, + DCNT_EP_POST_SEND_UD_DATA, + DCNT_EP_POST_RECV, + DCNT_EP_POST_RECV_DATA, + DCNT_EP_POST_WRITE, + DCNT_EP_POST_WRITE_DATA, + DCNT_EP_POST_WRITE_IMM, + DCNT_EP_POST_WRITE_IMM_DATA, + DCNT_EP_POST_READ, + DCNT_EP_POST_READ_DATA, + DCNT_EP_POST_CMP_SWAP, + DCNT_EP_POST_FETCH_ADD, + DCNT_EP_RECV, + DCNT_EP_RECV_DATA, + DCNT_EP_RECV_UD, + DCNT_EP_RECV_UD_DATA, + DCNT_EP_RECV_IMM, + DCNT_EP_RECV_IMM_DATA, + DCNT_EP_RECV_RDMA_IMM, + DCNT_EP_RECV_RDMA_IMM_DATA, + DCNT_EP_ALL_COUNTERS, /* MUST be last */ + +} DAT_EP_COUNTERS; + +/* + * Definitions for 64-bit EVD Counters + */ +typedef enum dat_evd_counters +{ + DCNT_EVD_WAIT, + DCNT_EVD_WAIT_BLOCKED, + DCNT_EVD_WAIT_NOTIFY, + DCNT_EVD_DEQUEUE, + DCNT_EVD_DEQUEUE_FOUND, + DCNT_EVD_DEQUEUE_NOT_FOUND, + DCNT_EVD_DEQUEUE_POLL, + DCNT_EVD_DEQUEUE_POLL_FOUND, + DCNT_EVD_CONN_CALLBACK, + DCNT_EVD_DTO_CALLBACK, + DCNT_EVD_ALL_COUNTERS, /* MUST be last */ + +} DAT_EVD_COUNTERS; /* Extended RETURN and EVENT STATUS string helper functions */ @@ -370,5 +472,32 @@ dat_strerror_ext_status ( (cookie), \ (flgs)) + +/* + * Query counter(s): + * Provide IA, EP, or EVD and call will return appropriate counters + * DAT_HANDLE dat_handle, enum cntr, *DAT_UINT64 p_cntrs_out, int reset + * + * use _ALL_COUNTERS to query all + */ +#define dat_query_counters(dat_handle, cntr, p_cntrs_out, reset) \ + dat_extension_op( dat_handle, \ + DAT_QUERY_COUNTERS_OP, \ + (cntr), \ + (p_cntrs_out), \ + (reset)) +/* + * Print counter(s): + * Provide IA, EP, or EVD and call will print appropriate counters + * DAT_HANDLE dat_handle, int cntr, int reset + * + * use _ALL_COUNTERS to print all + */ +#define dat_print_counters(dat_handle, cntr, reset) \ + dat_extension_op( dat_handle, \ + DAT_PRINT_COUNTERS_OP, \ + (cntr), \ + (reset)) + #endif /* _DAT_IB_EXTENSIONS_H_ */ diff --git a/test/dtest/dtestx.c b/test/dtest/dtestx.c index 032cadd..eafc1c9 100755 --- a/test/dtest/dtestx.c +++ b/test/dtest/dtestx.c @@ -60,6 +60,7 @@ #define DAPL_PROVIDER "ofa-v2-ib0" #define F64x "%"PRIx64"" +#define F64u "%"PRIu64"" #endif @@ -74,6 +75,7 @@ int disconnect_ep(void); if (status != DAT_SUCCESS) {\ dat_strerror(status, &maj_msg, &min_msg);\ fprintf(stderr, str " returned %s : %s\n", maj_msg, min_msg);\ + dat_ia_close(ia, DAT_CLOSE_DEFAULT);\ exit(1);\ } else if (verbose) {\ printf("dtestx: %s success\n",str);\ @@ -86,6 +88,7 @@ int disconnect_ep(void); if (status != DAT_SUCCESS) {\ dat_strerror(status, &maj_msg, &min_msg);\ fprintf(stderr, str " returned %s : %s\n", maj_msg, min_msg);\ + dat_ia_close(ia, DAT_CLOSE_DEFAULT);\ exit(1);\ } else if (verbose) {\ printf("dtestx: %s\n",str);\ @@ -162,6 +165,7 @@ DAT_EVD_HANDLE con_evd = DAT_HANDLE_NULL; DAT_EVD_HANDLE dto_evd = DAT_HANDLE_NULL; DAT_PSP_HANDLE psp = DAT_HANDLE_NULL; int server = 1; +int remote_host = 0; int ud_test = 0; int multi_eps = 0; int buf_size = BUF_SIZE; @@ -171,6 +175,8 @@ char hostname[256] = { 0 }; DAT_IB_ADDR_HANDLE remote_ah[MAX_EP_COUNT]; int eps = 1; int verbose = 0; +int counters = 0; +int counters_ok = 0; #define LOGPRINTF if (verbose) printf @@ -178,11 +184,12 @@ void print_usage(void) { printf("\n dtestx usage \n\n"); printf("v: verbose\n"); + printf("p: print counters\n"); printf("u unreliable datagram test\n"); printf("U: unreliable datagram test, UD endpoint count\n"); printf("m unreliable datagram test, multiple Server endpoints\n"); printf("b: buf length to allocate\n"); - printf("h: hostname/address of Server, specified on Client\n"); + printf("h: hostname/address of Server, client and UDP server\n"); printf("c: Client\n"); printf("s: Server, default\n"); printf("P: provider name (default = ofa-v2-ib0)\n"); @@ -400,6 +407,7 @@ connect_ep(char *hostname) DAT_RMR_TRIPLET *r_iov; DAT_DTO_COOKIE cookie; DAT_CONN_QUAL conn_qual; + DAT_BOOLEAN in,out; int i,ii,pdata,ctx; DAT_PROVIDER_ATTR prov_attrs; DAT_DTO_COMPLETION_EVENT_DATA *dto_event = @@ -418,6 +426,19 @@ connect_ep(char *hostname) LOGPRINTF(" Provider Specific Attribute[%d] %s=%s\n", i, prov_attrs.provider_specific_attr[i].name, prov_attrs.provider_specific_attr[i].value); + + /* check for counter support */ + status = strcmp(prov_attrs.provider_specific_attr[i].name, + "DAT_COUNTERS"); + if (!status) + counters_ok = 1; + } + + /* make sure provider supports counters */ + if ((counters) && (!counters_ok)) { + printf("Disable dat_query_counters:" + " Provider not built with counters\n"); + counters = 0; } status = dat_pz_create(ia, &pz); @@ -632,6 +653,9 @@ connect_ep(char *hostname) cookie, DAT_COMPLETION_SUPPRESS_FLAG); + dat_ep_get_status(ep[0],NULL,&in,&out); + printf("EP[0] status: posted buffers: Req=%d, Rcv=%d\n",in,out); + /* * Wait for their RMR */ @@ -730,9 +754,49 @@ disconnect_ep(void) _OK2(status, "dat_lmr_free_atomic"); } for (i=0;i add missing F64u definition for windows Signed-off-by: Arlin Davis --- test/dtest/dtestx.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/test/dtest/dtestx.c b/test/dtest/dtestx.c index eafc1c9..2fa0fc3 100755 --- a/test/dtest/dtestx.c +++ b/test/dtest/dtestx.c @@ -45,6 +45,7 @@ #define __BYTE_ORDER __LITTLE_ENDIAN #define F64x "%I64x" +#define F64u "%I64u" #define DAPL_PROVIDER "ibnic0v2" #else #include -- 1.5.2.5 From sean.hefty at intel.com Tue Mar 3 13:15:59 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 3 Mar 2009 13:15:59 -0800 Subject: [ofa-general] [PATCH] [ib-diag] ibstat: port_dump is called too many times Message-ID: <40F59C0D0E0E47F99E3D21458877BF31@amr.corp.intel.com> portnum index should go from 0 to numports - 1. This fixes an issue on WinOF where a bogus last port is displayed. Signed-off-by: Sean Hefty --- infiniband-diags/src/ibstat.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/infiniband-diags/src/ibstat.c b/infiniband-diags/src/ibstat.c index 7985be1..41faea8 100644 --- a/infiniband-diags/src/ibstat.c +++ b/infiniband-diags/src/ibstat.c @@ -146,7 +146,7 @@ ca_stat(char *ca_name, int portnum, int no_ports) if (no_ports) return 0; - for (portnum = 0; portnum <= ca.numports; portnum++) + for (portnum = 0; portnum < ca.numports; portnum++) port_dump(ca.ports[portnum], 0); return 0; From andy.grover at gmail.com Tue Mar 3 14:19:34 2009 From: andy.grover at gmail.com (Andrew Grover) Date: Tue, 3 Mar 2009 14:19:34 -0800 Subject: ***SPAM*** Re: [ofa-general] [PATCH 0/26] Reliable Datagram Sockets (RDS), take 2 In-Reply-To: <20090301015020.GH26292@one.firstfloor.org> References: <1235525443-9007-1-git-send-email-andy.grover@oracle.com> <87myc73izx.fsf@basil.nowhere.org> <20090228055608.GB26292@one.firstfloor.org> <20090228223653.GD26292@one.firstfloor.org> <20090301015020.GH26292@one.firstfloor.org> Message-ID: On Sat, Feb 28, 2009 at 5:50 PM, Andi Kleen wrote: >> Higher load -> more time before a process runs -> rcvbuf overfills -> > > How can the rcvbuf overfill if the sender doesn't run? rcvbuf overfill on the receiver-side. -- Andy From rdreier at cisco.com Tue Mar 3 14:22:26 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 03 Mar 2009 14:22:26 -0800 Subject: [ofa-general] Re: [PATCH] IB/core: ib_post_send_mad() returns zero but doesn't generate send completion In-Reply-To: <1235771157.3948.233.camel@chromite.mv.qlogic.com> (Ralph Campbell's message of "Fri, 27 Feb 2009 13:45:57 -0800") References: <1235771157.3948.233.camel@chromite.mv.qlogic.com> Message-ID: thanks, applied From rdreier at cisco.com Tue Mar 3 14:35:12 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 03 Mar 2009 14:35:12 -0800 Subject: [ofa-general] Re: [PATCH] IB/sa_query: fix update_sm_ah() race condition. In-Reply-To: <200903030920.46659.jackm@dev.mellanox.co.il> (Jack Morgenstein's message of "Tue, 3 Mar 2009 09:20:46 +0200") References: <200903030920.46659.jackm@dev.mellanox.co.il> Message-ID: thanks, applied. From Jie.Cai at cs.anu.edu.au Tue Mar 3 18:35:36 2009 From: Jie.Cai at cs.anu.edu.au (Jie Cai) Date: Wed, 04 Mar 2009 13:35:36 +1100 Subject: [ofa-general] Bandwidth of performance with multirail IB In-Reply-To: References: <20090223211155.730AFE28137@openfabrics.org> <49A378BC.5010806@cs.anu.edu.au> <200902240941.58634.cap@nsc.liu.se> <49A7885B.3010005@cs.anu.edu.au> Message-ID: <49ADE8F8.8030709@cs.anu.edu.au> Davis, Arlin R wrote: > > Jie, > > >> A question on implementation multi-rail with uDAPL connections. >> >> What I did is open 2 IAs (corresponding to the 2 ports on >> HCAs) on each >> node. >> Then create one EP for each IA, and connect those EPs to the >> corresponding >> EP at other node. >> >> Then data been transferred via both EP-connections. >> > > Did you verify data flow is across independent HCA channels > using perfquery? > > > Don't know about perfquery, will use this to verify today. However, I used a rather simple way to verify the both rail has been used for data transferring. The both rail writes to different remote buffer, and the written data has been confirmed at the remote side. >> I have been notice that there's a MULTIPATH connection flag for dapl, >> but I did >> not use it. What's the use of it? >> > > Multipath is not supported at this time. > > I will roll up a uDAPL test and see if I can verify performance. > Thanks for this. > -arlin > > From zhouyonghao at ict.ac.cn Tue Mar 3 22:43:32 2009 From: zhouyonghao at ict.ac.cn (zhouyonghao at ict.ac.cn) Date: Wed, 4 Mar 2009 14:43:32 +0800 (CST) Subject: [ofa-general] puzzled by ATOMIC operation Message-ID: <62a06b6fde0e9edc1fef0abc31a89abc.squirrel@webmail.ict.ac.cn> Hi, all these days I'm puzzled by a weird mistake in atomic operation, when I use FETCH_AND_ADD, the remote buffer only changed for 9 times at most, aka if the initial value is 0, when I execute 10 times FETCH_AND_ADD operations, the value is 9, and no more changes happens to the buffer. Should I take care of the big-edian from/to litter-edian translation? I tried this, but it seems not the point. Is there some sample code for atomic operation just like "perftest" in the libibverbs package? Maybe I missed something? I paste some code below for reviewing, thanks in advance! struct ibv_send_wr wr; wr.opcode = IBV_WR_ATOMIC_FETCH_AND_ADD; wr.wr.atomic.remote_addr = rem_dest->vaddr; wr.wr.atomic.rkey = rem_dest->rkey; wr.send_flags = IBV_SEND_SIGNALED | IBV_SEND_FENCE; wr.wr.atomic.compare_add = 1; wr.wr.atomic.swap = 0; wr.wr_id = id; wr.sg_list = &list; wr.num_sge = 1; wr.next = NULL; the qp and memory region is set with atomic flag : mr = ibv_reg_mr(host->pd, buf, len, IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_ATOMIC); attr.qp_access_flags = IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_ATOMIC; some attibutes in my ib device about atomic operation are as below: max_qp_rd_atom is 4, max_qp_init_rd_atom is 128, max_res_rd_atom is 258048 From chenggangschool at yahoo.com.cn Tue Mar 3 23:19:01 2009 From: chenggangschool at yahoo.com.cn (=?utf-8?B?5byg5ZGI5Yia?=) Date: Wed, 4 Mar 2009 15:19:01 +0800 (CST) Subject: [ofa-general] OFED-1.4 build problems on Redhat Enterprise 5 Message-ID: <746630.74312.qm@web15708.mail.cnb.yahoo.com> Hi, all !   I'm trying to compile OFED-1.4 on Red Hat Enterprise 5 (the kernel is 2.6.18-8.el5xen, ps:the CPU arch is i686). I have installed all the dependencies listed in readme.txt.           I followed the OFED_Install_Guide.txt(using ./install.pl  in the OFED directory) and could not get it installed though.   Below is the errors from the log. Also, the whole ofa_kernel.rpmbuild.log  is attached. I am quite new to this and hope someone can help me. Does anyone know how to fix this?  Thanks in advance!     Processing files: kernel-ib-1.4-2.6.18_8.el5xen error: File not found: /var/tmp/OFED/lib/modules/2.6.18-8.el5xen/updates/kernel/drivers/scsi/iscsi_tcp.ko error: File not found: /var/tmp/OFED/lib/modules/2.6.18-8.el5xen/updates/kernel/drivers/scsi/libiscsi.ko error: File not found: /var/tmp/OFED/lib/modules/2.6.18-8.el5xen/updates/kernel/drivers/scsi/scsi_transport_iscsi.ko error: File not found: /var/tmp/OFED/lib/modules/2.6.18-8.el5xen/updates/kernel/net/rds error: File not found: /var/tmp/OFED/lib/modules/2.6.18-8.el5xen/updates/kernel/drivers/net/cxgb3 error: File not found: /var/tmp/OFED/lib/modules/2.6.18-8.el5xen/updates/kernel/drivers/net/mlx4 Processing files: kernel-ib-devel-1.4-2.6.18_8.el5xen Requires(interp): /bin/sh /bin/sh /bin/sh Requires(rpmlib): rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1 Requires(pre): /bin/sh Requires(post): /bin/sh Requires(postun): /bin/sh Requires: /bin/bash kernel-ib RPM build errors:     user vlad does not exist - using root     group vlad does not exist - using root     user vlad does not exist - using root     group vlad does not exist - using root     File not found: /var/tmp/OFED/lib/modules/2.6.18-8.el5xen/updates/kernel/drivers/scsi/iscsi_tcp.ko     File not found: /var/tmp/OFED/lib/modules/2.6.18-8.el5xen/updates/kernel/drivers/scsi/libiscsi.ko     File not found: /var/tmp/OFED/lib/modules/2.6.18-8.el5xen/updates/kernel/drivers/scsi/scsi_transport_iscsi.ko     File not found: /var/tmp/OFED/lib/modules/2.6.18-8.el5xen/updates/kernel/net/rds     File not found: /var/tmp/OFED/lib/modules/2.6.18-8.el5xen/updates/kernel/drivers/net/cxgb3     File not found: /var/tmp/OFED/lib/modules/2.6.18-8.el5xen/updates/kernel/drivers/net/mlx4   Regards, Bob     ___________________________________________________________ 好玩贺卡等你发，邮箱贺卡全新上线！ http://card.mail.cn.yahoo.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ofa_kernel.rpmbuild.log Type: application/octet-stream Size: 1334549 bytes Desc: not available URL: From nicolas.morey-chaisemartin at ext.bull.net Wed Mar 4 00:58:53 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Wed, 04 Mar 2009 09:58:53 +0100 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_switch.h : Fixed wrong comment about return value of osm_switch_set_hops Message-ID: <49AE42CD.5070802@ext.bull.net> Signed-off-by: Nicolas Morey-Chaisemartin --- Comment might have been copied/pasted to fast from osm_switch_get_hops opensm/include/opensm/osm_switch.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h index 3e3626b..0b15e63 100644 --- a/opensm/include/opensm/osm_switch.h +++ b/opensm/include/opensm/osm_switch.h @@ -297,7 +297,7 @@ osm_switch_set_hops(IN osm_switch_t * const p_sw, * [in] value to assign to this entry. * * RETURN VALUES -* Returns the hop count at the specified LID/Port intersection. +* Returns 0 if successfull. -1 if it failed * * NOTES * -- 1.6.2-rc2.GIT From Line.Holen at Sun.COM Wed Mar 4 01:27:10 2009 From: Line.Holen at Sun.COM (Line.Holen at Sun.COM) Date: Wed, 04 Mar 2009 10:27:10 +0100 Subject: [ofa-general] [PATCH] opensm/osm_link_mgr.c initialize SMSL Message-ID: <49AE496E.1040606@Sun.COM> The SMSL is currently not initialized. Whatever is written in PortInfo at power up is kept. The lash routing algorithm makes active use of SL to avoid deadlock in the fabric. The endnodes needs to be instructed to use the SL calculated for the endnode-to-SMnode path for SA traffic. If not, the SA traffic can cause deadlock. This change makes sure that the SMSL is properly initialized on all ports. If flash routing is used, the SMSL is set up based on routing calculations. For other routing algorithms the default SMSL is used. Signed-off-by: Line Holen --- diff --git a/opensm/opensm/osm_link_mgr.c b/opensm/opensm/osm_link_mgr.c index 37e3e1b..44765e3 100644 --- a/opensm/opensm/osm_link_mgr.c +++ b/opensm/opensm/osm_link_mgr.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -51,6 +52,49 @@ #include #include #include +#include + +extern uint8_t osm_get_lash_sl(osm_opensm_t * p_osm, + const osm_port_t * p_src_port, + const osm_port_t * p_dst_port); + +/********************************************************************** + **********************************************************************/ +static uint8_t +__osm_link_mgr_get_smsl(IN osm_sm_t * sm, + IN osm_physp_t * const p_physp) +{ + osm_opensm_t *p_osm = sm->p_subn->p_osm; + const osm_port_t *p_sm_port; + const osm_port_t *p_src_port; + ib_net16_t slid; + ib_net16_t smlid; + uint8_t sl; + + OSM_LOG_ENTER(sm->p_log); + + if (p_osm->routing_engine_used != OSM_ROUTING_ENGINE_TYPE_LASH) { + /* Use default SL if lash routing is not used */ + OSM_LOG_EXIT(sm->p_log); + return (OSM_DEFAULT_SL); + } + + /* Find osm_port of the SM itself = dest_port */ + smlid = sm->p_subn->sm_base_lid; + p_sm_port = cl_ptr_vector_get(&sm->p_subn->port_lid_tbl, cl_ntoh16(smlid)); + + /* Find osm_port of the source = p_physp */ + slid = osm_physp_get_base_lid(p_physp); + p_src_port = cl_ptr_vector_get(&sm->p_subn->port_lid_tbl, cl_ntoh16(slid)); + + /* Call lash to find proper SL */ + sl = osm_get_lash_sl(p_osm, p_src_port, p_sm_port); + OSM_LOG(sm->p_log, OSM_LOG_DEBUG, "SL for LID %u to SA is %d\n", + cl_ntoh16(slid), sl); + + OSM_LOG_EXIT(sm->p_log); + return (sl); +} /********************************************************************** **********************************************************************/ @@ -67,6 +111,7 @@ __osm_link_mgr_set_physp_pi(osm_sm_t * sm, ib_api_status_t status; uint8_t port_num; uint8_t mtu; + uint8_t smsl; uint8_t op_vls; boolean_t esp0 = FALSE; boolean_t send_set = FALSE; @@ -76,6 +121,8 @@ __osm_link_mgr_set_physp_pi(osm_sm_t * sm, p_node = osm_physp_get_node_ptr(p_physp); + p_old_pi = &p_physp->port_info; + port_num = osm_physp_get_port_num(p_physp); if (port_num == 0) { @@ -93,8 +140,40 @@ __osm_link_mgr_set_physp_pi(osm_sm_t * sm, if (ib_switch_info_is_enhanced_port0(&p_node->sw->switch_info) == FALSE) { - /* This means the switch doesn't support enhanced port 0. - Can skip it. */ + + /* Even for base port 0 we might have to set smsl + (if we are using lash routing) */ + smsl = __osm_link_mgr_get_smsl(sm, p_physp); + if (smsl != ib_port_info_get_master_smsl(p_old_pi)) { + + OSM_LOG(sm->p_log, OSM_LOG_DEBUG, + "Setting SMSL to %d on base port0 switch 0x%016" + PRIx64 "\n", + smsl, + cl_ntoh64(osm_physp_get_port_guid(p_physp))); + + memset(payload, 0, IB_SMP_DATA_SIZE); + memcpy(payload, p_old_pi, sizeof(ib_port_info_t)); + + /* + Should never write back a value that is bigger than 3 in + the PortPhysicalState field - so can not simply copy! + + Actually we want to write there: port physical state + - no change, link down default state = polling + port state - as requested. + */ + p_pi->state_info2 = 0x02; + ib_port_info_set_port_state(p_pi, port_state); + + ib_port_info_set_master_smsl(p_pi, smsl); + + send_set = TRUE; + goto Send; + } + + /* This means the switch doesn't support enhanced port 0 + and we don't need to change SMSL. Can skip it. */ OSM_LOG(sm->p_log, OSM_LOG_DEBUG, "Skipping port 0, GUID 0x%016" PRIx64 "\n", cl_ntoh64(osm_physp_get_port_guid(p_physp))); @@ -107,8 +186,6 @@ __osm_link_mgr_set_physp_pi(osm_sm_t * sm, PAST THIS POINT WE ARE HANDLING EITHER A NON PORT 0 OR ENHANCED PORT 0 */ - p_old_pi = &p_physp->port_info; - memset(payload, 0, IB_SMP_DATA_SIZE); memcpy(payload, p_old_pi, sizeof(ib_port_info_t)); @@ -160,6 +237,20 @@ __osm_link_mgr_set_physp_pi(osm_sm_t * sm, sizeof(p_pi->master_sm_base_lid))) send_set = TRUE; + smsl = __osm_link_mgr_get_smsl(sm, p_physp); + if (smsl != ib_port_info_get_master_smsl(p_old_pi)) { + + ib_port_info_set_master_smsl(p_pi, smsl); + + OSM_LOG(sm->p_log, OSM_LOG_DEBUG, + "Setting SMSL to %d on GUID 0x%016" + PRIx64 ", port %d\n", + smsl, cl_ntoh64(osm_physp_get_port_guid(p_physp)), + port_num); + + send_set = TRUE; + } + p_pi->m_key_lease_period = sm->p_subn->opt.m_key_lease_period; if (memcmp(&p_pi->m_key_lease_period, @@ -290,6 +381,7 @@ __osm_link_mgr_set_physp_pi(osm_sm_t * sm, } } +Send: if (port_state != IB_LINK_NO_CHANGE && port_state != ib_port_info_get_port_state(p_old_pi)) { send_set = TRUE; From olga.shern at gmail.com Wed Mar 4 01:48:21 2009 From: olga.shern at gmail.com (Olga Shern (Voltaire)) Date: Wed, 4 Mar 2009 11:48:21 +0200 Subject: ***SPAM*** Re: [ofa-general] OFED-1.4 build problems on Redhat Enterprise 5 In-Reply-To: <746630.74312.qm@web15708.mail.cnb.yahoo.com> References: <746630.74312.qm@web15708.mail.cnb.yahoo.com> Message-ID: Hi Bob > I'm trying to compile OFED-1.4 on Red Hat Enterprise 5 (the kernel is > 2.6.18-8.el5xen, ps:the CPU arch is i686). I have installed all the > dependencies listed in readme.txt. xen kernel is not supported by OFED, you should install 2.6.18.8-EL5 kernel Olga From kliteyn at dev.mellanox.co.il Wed Mar 4 02:15:27 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Wed, 04 Mar 2009 12:15:27 +0200 Subject: [ofa-general] [RFC] Fat-Tree upgrades In-Reply-To: <498FDDD5.1090204@ext.bull.net> References: <498FDDD5.1090204@ext.bull.net> Message-ID: <49AE54BF.3080000@dev.mellanox.co.il> Nicolas Morey Chaisemartin wrote: > Hi everyone, > > We have been working quite a lot at Bull lately on the Ftree algorithm > and we have made some upgrades. > However, as they modify the behavior of the ftree algorithm, we haven't > pushed them until now. > I'm just going to detail which upgrades we have done and let you decide > if you are interested, if and how they should be pushed upstream (new > routing algorithm, option in the ftree, etc.) > > Here is a simplify model of the topology we have been working on > > L3 L3 > ___________________|__|____________________ > / / \ \ <= All > the L2 are connected on 2 L3 switches > L2-1 L2-2 L2-1 L2-2 <= > There are service nodes connected directly on L2 switches > / \S1 / \S2 S3/ \ S4/ \ <== > The Nth L1 of a set leads only to the Nth L2 (L2-N). With some pruning. > L1 L1 L1 L1 > /|\ /|\ /|\ /|\ > ==Fully mixed to L1== ==Fully mixed to L1== <=== We have > multiple set. In each set, all L0 lead to all L1 of their set. > > L0 L0 L0 L0 > / \ / \ / \ / \ > CN CN .. CN CN .... CN CN .. CN CN > > To detail: > We have a bunch of sets. Each set contains compute node, L0 and L1 > switches. > Plus a common top of L2 and L3 switches. > > In each set, there are groups of compute nodes. Each group is connected > to a single L0 switch. > In a given set, all L0 are connected to all L1. > > The Nth L1 of a set is connected to the Nth L2 and only to this one. (so > through a L2, the Nth L1 can only see the Nth L1 of the other sets) > There are Services nodes connected to the L2 switches. > All the L2 are connected to a couple of L3. > > The problem we have seen when routing on this topology is that most of > the routes from CN to SN (service nodes) go through the L3 switches. > With the current algorithm, the less loaded link is choosed to go down > by going up. Therefore, the primary path goes through a L2, then a L3 > from where it covers all the network. Not sure I understand why. There are two stages of routing: 1. Route the downward paths, which is done by climbing up the tree. 2. Route the upward paths, which is done by descending. Before each climbing, the routing first configures all the upward paths: the first acll in the beginning of the __osm_ftree_fabric_route_downgoing_by_going_up() function is __osm_ftree_fabric_route_upgoing_by_going_down(). So in your example, if some Service Node (SN) is connected to L2-i switch, the routing should first configure all the i'th switches in the L1 sets, and then all the L0 switches in all the sets because L1 is fully mixed to L0 in each set. Where am I wrong? > This wasn't acceptable for us as L3 switches would be overloaded when > there were less loaded/shorter paths to achieve the same HCA. > > So what we have introduced here is a "balanced min_hop" within the ftree > algorithm. > Basically, instead of just leaving when we reach a LFT which has already > been configured for the target lid, we check the hops count of the > switch toward this lid, and the hop count on the path we came through. > If we have found a shorter path, we update the LFT and minhop tables to > use this new path. I don't think that fat-tree routing was supposed to configure a route with more hops that min-hop would do it. Either I don't understand the example, or there is a bug in the routing. -- Yevgeny > This means that the difference between primary_path and secondary path > is not so important anymore. > Secondary path may increment port counters but only if routes to HCA > were created (see opensm/osm_ucast_ftree.c: Fixed bug on index port > incrementation which makes this possible). > I acknowledge that port count may be slightly wrong as a primary path > that is replaced with a shorter secondary path has incremented counters > and they won't be removed. However, in most the cases the primary path > would have created other routes than the one replaced so counters are fine. > > For all regular ftree topology, I have see no change with this update > but with topologies where two levels are not fully interconnected, this > helps a lot ! > > > > Another thing we have developped here is to balance more secondary path. > In the current algorithm, secondary down path (going_down_by_going up) > are created in port_group order. > This means that if the primary path didn't reach all the network > (because a switch is broken for examples), all the routes missing will > be created through the first port group. Which unbalance the network > load a lot. > To solve this, we create the secondary path by port group load. > The previous patch has made us increment the port/portgroup counters > when secondary routes towards HCA are created, therefore these counters > are significant even when creating secondary routes. > What our patch does is at the beginning of the function sort all the > port group from lowest load to highest. Pick the first one for the > primary path, and try secondary path from the 2nd to the last. > Once again this seems to have no effect on regular topology but it made > a real impact on our failover tests. > > > > Feel free to comment this, and more specially if and how you would want > them upstream. > > Thanks in advance > > Nicolas > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From vlad at lists.openfabrics.org Wed Mar 4 03:20:55 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Wed, 4 Mar 2009 03:20:55 -0800 (PST) Subject: [ofa-general] ofa_1_4_kernel 20090304-0200 daily build status Message-ID: <20090304112055.D3EC5E60E78@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From nicolas.morey-chaisemartin at ext.bull.net Wed Mar 4 03:45:56 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Wed, 04 Mar 2009 12:45:56 +0100 Subject: [ofa-general] [RFC] Fat-Tree upgrades In-Reply-To: <49AE54BF.3080000@dev.mellanox.co.il> References: <498FDDD5.1090204@ext.bull.net> <49AE54BF.3080000@dev.mellanox.co.il> Message-ID: <49AE69F4.2080903@ext.bull.net> Yevgeny Kliteynik wrote: > Nicolas Morey Chaisemartin wrote: >> Hi everyone, >> >> We have been working quite a lot at Bull lately on the Ftree algorithm >> and we have made some upgrades. >> However, as they modify the behavior of the ftree algorithm, we >> haven't pushed them until now. >> I'm just going to detail which upgrades we have done and let you >> decide if you are interested, if and how they should be pushed >> upstream (new routing algorithm, option in the ftree, etc.) >> >> Here is a simplify model of the topology we have been working on >> >> L3 L3 >> ___________________|__|____________________ >> / / \ \ <= >> All the L2 are connected on 2 L3 switches >> L2-1 L2-2 L2-1 L2-2 <= >> There are service nodes connected directly on L2 switches >> / \S1 / \S2 S3/ \ S4/ \ <== >> The Nth L1 of a set leads only to the Nth L2 (L2-N). With some pruning. >> L1 L1 L1 L1 >> /|\ /|\ /|\ /|\ >> ==Fully mixed to L1== ==Fully mixed to L1== <=== We have >> multiple set. In each set, all L0 lead to all L1 of their set. >> >> L0 L0 L0 L0 >> / \ / \ / \ / \ >> CN CN .. CN CN .... CN CN .. CN CN >> >> To detail: >> We have a bunch of sets. Each set contains compute node, L0 and L1 >> switches. >> Plus a common top of L2 and L3 switches. >> >> In each set, there are groups of compute nodes. Each group is >> connected to a single L0 switch. >> In a given set, all L0 are connected to all L1. >> >> The Nth L1 of a set is connected to the Nth L2 and only to this one. >> (so through a L2, the Nth L1 can only see the Nth L1 of the other sets) >> There are Services nodes connected to the L2 switches. >> All the L2 are connected to a couple of L3. >> >> The problem we have seen when routing on this topology is that most of >> the routes from CN to SN (service nodes) go through the L3 switches. >> With the current algorithm, the less loaded link is choosed to go down >> by going up. Therefore, the primary path goes through a L2, then a L3 >> from where it covers all the network. > > Not sure I understand why. > There are two stages of routing: > 1. Route the downward paths, which is done by climbing up the tree. > 2. Route the upward paths, which is done by descending. > Before each climbing, the routing first configures all the upward > paths: the first acll in the beginning of the > __osm_ftree_fabric_route_downgoing_by_going_up() function is > __osm_ftree_fabric_route_upgoing_by_going_down(). OK > > So in your example, if some Service Node (SN) is connected to L2-i > switch, the routing should first configure all the i'th switches > in the L1 sets, and then all the L0 switches in all the sets because > L1 is fully mixed to L0 in each set. > > Where am I wrong? You're right. Routes towards the service nodes are OK. Problem is the routes from the service node. Let's take a CN (called C0) attached to a L0. Algorithm go one step up (downward route) to L1-1 (could be any L1 in its set), covers all the upward routes so we have all the routes CN->C0 where the CN belong to the same set. Next step: Algorithm doesn't have a choice here, it goes to L2-1, covers the upward routes so we have all the CN->C0 routes and S1->C0 However we don't have any routes from S[234]->C0 yet. Next and final step, algorithm go to either one of the L3, covers upwards routes and create routes from S[234]->C0. These routes have more hops than the minimum possible. > >> This wasn't acceptable for us as L3 switches would be overloaded when >> there were less loaded/shorter paths to achieve the same HCA. >> >> So what we have introduced here is a "balanced min_hop" within the >> ftree algorithm. >> Basically, instead of just leaving when we reach a LFT which has >> already been configured for the target lid, we check the hops count of >> the switch toward this lid, and the hop count on the path we came >> through. If we have found a shorter path, we update the LFT and minhop >> tables to use this new path. > > I don't think that fat-tree routing was supposed to configure a route > with more hops that min-hop would do it. Either I don't understand the > example, or there is a bug in the routing. If it isn't suppose to, then it's a bug :) However this bug only happens with nodes not at the same level as compute nodes. > > -- Yevgeny > From hnrose at comcast.net Wed Mar 4 07:12:28 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 4 Mar 2009 10:12:28 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] opensm/osm_req.c: In osm_send_trap144, set producer type according to node type Message-ID: <20090304151228.GB3820@comcast.net> rather than always set to CA Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c index 0865ce5..2e851b2 100644 --- a/opensm/opensm/osm_req.c +++ b/opensm/opensm/osm_req.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -261,7 +262,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) ntc = (ib_mad_notice_attr_t *)smp->data; ntc->generic_type = 0x80 | IB_NOTICE_TYPE_INFO; - ib_notice_set_prod_type_ho(ntc, IB_NODE_TYPE_CA); + ib_notice_set_prod_type_ho(ntc, osm_node_get_type(port->p_node)); ntc->g_or_v.generic.trap_num = cl_hton16(144); ntc->issuer_lid = pi->base_lid; ntc->data_details.ntc_144.lid = pi->base_lid; From hnrose at comcast.net Wed Mar 4 07:11:38 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 4 Mar 2009 10:11:38 -0500 Subject: [ofa-general] opensm/libvendor/osm_vendor_ibumad.c: In clear_madw, fix tid endian in message Message-ID: <20090304151137.GA3820@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/libvendor/osm_vendor_ibumad.c b/opensm/libvendor/osm_vendor_ibumad.c index 734a860..a2e15f4 100644 --- a/opensm/libvendor/osm_vendor_ibumad.c +++ b/opensm/libvendor/osm_vendor_ibumad.c @@ -112,7 +112,7 @@ static void clear_madw(osm_vendor_t * p_vend) pthread_mutex_unlock(&p_vend->match_tbl_mutex); OSM_LOG(p_vend->p_log, OSM_LOG_ERROR, "ERR 5401: " "evicting entry %p (tid was 0x%" PRIx64 ")\n", - old_m, old_tid); + old_m, cl_ntoh64(old_tid)); goto Exit; } } From hal.rosenstock at gmail.com Wed Mar 4 07:24:54 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 4 Mar 2009 10:24:54 -0500 Subject: [ofa-general] ***SPAM*** opensm send_trap144 question Message-ID: Sasha, In commits 663255db91ecd2baf78b28d77b4397314e5349ca and e59489bf06bfed7491b7c3da64ca22ed326b1563, there is the potential for trap 144 to be generated. We've discussed some aspects of this when the patches were first on the list. There's another aspect of this that comes to light now that I've started playing with this. In osm_req.c, osm_send_trap144 does not set (or for that matter currently clear) the resp_expected flag. It seems to me that a more robust strategy would be to set resp_expected to try to make sure that the other side sees the trap (and the timeout/retry strategy is used). There are a number of underpinnings to change to do this. Would you accept a patch along those lines ? -- Hal From kliteyn at dev.mellanox.co.il Wed Mar 4 07:37:26 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Wed, 04 Mar 2009 17:37:26 +0200 Subject: [ofa-general] [RFC] Fat-Tree upgrades In-Reply-To: <49AE69F4.2080903@ext.bull.net> References: <498FDDD5.1090204@ext.bull.net> <49AE54BF.3080000@dev.mellanox.co.il> <49AE69F4.2080903@ext.bull.net> Message-ID: <49AEA036.2090005@dev.mellanox.co.il> Hi Nicolas, Nicolas Morey Chaisemartin wrote: > Yevgeny Kliteynik wrote: >> Nicolas Morey Chaisemartin wrote: >>> Hi everyone, >>> >>> We have been working quite a lot at Bull lately on the Ftree algorithm >>> and we have made some upgrades. >>> However, as they modify the behavior of the ftree algorithm, we >>> haven't pushed them until now. >>> I'm just going to detail which upgrades we have done and let you >>> decide if you are interested, if and how they should be pushed >>> upstream (new routing algorithm, option in the ftree, etc.) >>> >>> Here is a simplify model of the topology we have been working on >>> >>> L3 L3 >>> ___________________|__|____________________ >>> / / \ \ <= >>> All the L2 are connected on 2 L3 switches >>> L2-1 L2-2 L2-1 L2-2 <= >>> There are service nodes connected directly on L2 switches >>> / \S1 / \S2 S3/ \ S4/ \ <== >>> The Nth L1 of a set leads only to the Nth L2 (L2-N). With some pruning. >>> L1 L1 L1 L1 >>> /|\ /|\ /|\ /|\ >>> ==Fully mixed to L1== ==Fully mixed to L1== <=== We have >>> multiple set. In each set, all L0 lead to all L1 of their set. >>> >>> L0 L0 L0 L0 >>> / \ / \ / \ / \ >>> CN CN .. CN CN .... CN CN .. CN CN >>> >>> To detail: >>> We have a bunch of sets. Each set contains compute node, L0 and L1 >>> switches. >>> Plus a common top of L2 and L3 switches. >>> >>> In each set, there are groups of compute nodes. Each group is >>> connected to a single L0 switch. >>> In a given set, all L0 are connected to all L1. >>> >>> The Nth L1 of a set is connected to the Nth L2 and only to this one. >>> (so through a L2, the Nth L1 can only see the Nth L1 of the other sets) >>> There are Services nodes connected to the L2 switches. >>> All the L2 are connected to a couple of L3. >>> >>> The problem we have seen when routing on this topology is that most of >>> the routes from CN to SN (service nodes) go through the L3 switches. >>> With the current algorithm, the less loaded link is choosed to go down >>> by going up. Therefore, the primary path goes through a L2, then a L3 >>> from where it covers all the network. >> Not sure I understand why. >> There are two stages of routing: >> 1. Route the downward paths, which is done by climbing up the tree. >> 2. Route the upward paths, which is done by descending. >> Before each climbing, the routing first configures all the upward >> paths: the first acll in the beginning of the >> __osm_ftree_fabric_route_downgoing_by_going_up() function is >> __osm_ftree_fabric_route_upgoing_by_going_down(). > > OK > >> So in your example, if some Service Node (SN) is connected to L2-i >> switch, the routing should first configure all the i'th switches >> in the L1 sets, and then all the L0 switches in all the sets because >> L1 is fully mixed to L0 in each set. >> >> Where am I wrong? > > You're right. Routes towards the service nodes are OK. > > Problem is the routes from the service node. > > Let's take a CN (called C0) attached to a L0. > Algorithm go one step up (downward route) to L1-1 (could be any L1 in its set), covers all the upward routes so we have all the routes CN->C0 where the CN belong to the same set. > Next step: Algorithm doesn't have a choice here, it goes to L2-1, covers the upward routes so we have all the CN->C0 routes and S1->C0 > However we don't have any routes from S[234]->C0 yet. > Next and final step, algorithm go to either one of the L3, covers upwards routes and create routes from S[234]->C0. These routes have more hops than the minimum possible. OK, I get it now. So there's is a general issue with the existing algorithm: in topologies with redundant switch level(s), such as L3 in your example, fat-tree might create switch-to-CN routes (or in your example, SN-to-CN when SN is on a higher tree level than CN) that are longer than min-hop. Currently, when the algorithm goes up, it checks whether or not the upper switch already has this LID in the LFT, and if it has - the route is abandoned. What you're suggesting is to check the path length, and if the new path is shorter, then it should replace the old one. Sounds like a great idea. >>> This wasn't acceptable for us as L3 switches would be overloaded when >>> there were less loaded/shorter paths to achieve the same HCA. >>> >>> So what we have introduced here is a "balanced min_hop" within the >>> ftree algorithm. >>> Basically, instead of just leaving when we reach a LFT which has >>> already been configured for the target lid, we check the hops count of >>> the switch toward this lid, and the hop count on the path we came >>> through. If we have found a shorter path, we update the LFT and minhop >>> tables to use this new path. And what do you do after that? Do you continue climbing the tree and checking the route length? I guess you need to continue climbing till all the upper switches will have LFT entries with route length equal to min-hop. I do have some concerns about the CN-to-CN routes in this case. Any chance these routes might be affected? My intuition says that they won't, because they are required to reside at the same tree level, and since there is full connectivity between them, they will be all configured with the shortest path. But I'm not done convincing myself... -- Yevgeny >> I don't think that fat-tree routing was supposed to configure a route >> with more hops that min-hop would do it. Either I don't understand the >> example, or there is a bug in the routing. > > If it isn't suppose to, then it's a bug :) > However this bug only happens with nodes not at the same level as compute nodes. >> -- Yevgeny >> > From tziporet at mellanox.co.il Wed Mar 4 08:06:58 2009 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Wed, 4 Mar 2009 18:06:58 +0200 Subject: [ofa-general] OFED 1.4.1 is available Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD01F50B50@mtlexch01.mtl.com> Hi, OFED-1.4.1-rc1 release is available on http://www.openfabrics.org/downloads/OFED/ofed-1.4.1/OFED-1.4.1-rc1.tgz To get BUILD_ID run ofed_info Please report any issues in bugzilla https://bugs.openfabrics.org/ for OFED 1.4.1 Vladimir & Tziporet ======================================================================== Release information: ------------------------------ Linux Operating Systems: - RedHat EL4 up4: 2.6.9-42.ELsmp * - RedHat EL4 up5: 2.6.9-55.ELsmp - RedHat EL4 up6: 2.6.9-67.ELsmp - RedHat EL4 up7: 2.6.9-78.ELsmp - RedHat EL5: 2.6.18-8.el5 - RedHat EL5 up1: 2.6.18-53.el5 - RedHat EL5 up2: 2.6.18-92.el5 - RedHat EL5 up3: 2.6.18-128.el5 - OEL 4.5: 2.6.9-55.ELsmp - OEL 5.2: 2.6.18-92.el5 - CentOS 5.2: 2.6.18-92.el5 - Fedora C9: 2.6.25-14.fc9 * - SLES10: 2.6.16.21-0.8-smp - SLES10 SP1: 2.6.16.46-0.12-smp - SLES10 SP1 up1: 2.6.16.53-0.16-smp - SLES10 SP2: 2.6.16.60-0.21-smp - SLES11 RC3: 2.6.27.13-1-default - OpenSuSE 10.3: 2.6.22.5-31 * - kernel.org: 2.6.26 and 2.6.27 * Minimal QA for these versions Systems: * x86_64 * x86 * ia64 * ppc64 Main Changes from OFED-1.4 ====================== - Added support for RHEL 5.3 and SLES11 RC3 - Added RHEL 5.2 backports for NFS/RDMA - Updated MPI packages: mvapich-1.1.0-3143 Open MPI 1.3.1-rc4 - Updated bonding package: ib-bonding-0.9.0-38 - Updated opensm version to include critical bug fixes - Fixed RDS iWARP support - Low level drivers updated: ehca, mlx4, cxgb3, nes - Bug fixes - Attached kernel git tree changes for details <> Tasks that should be completed for RC2: ============================= 1. High priority bug fixes 2. RHEL 5.2 32 bits compilation issue (due to NFS/RDMA backports) 3. Open MPI 1.3.1 release 4. MVAPICH 1.1.1.minor release with critical bug fixes 5. OSM release 3.2.6 6. Documentation update -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ofa_kernel-1.4_1.4.1-rc1.log Type: application/octet-stream Size: 83561 bytes Desc: ofa_kernel-1.4_1.4.1-rc1.log URL: From PHF at zurich.ibm.com Wed Mar 4 09:04:33 2009 From: PHF at zurich.ibm.com (Philip Frey1) Date: Wed, 4 Mar 2009 18:04:33 +0100 Subject: [ofa-general] NFS over RDMA Message-ID: Hello, I want to try NFS over RDMA. According to the documentation enclosed within the OFED-1.4 release, it should be as simple as setting "nfsrdma=y". Unfortunately there is no such option. The interactive menu offered by './install.pl' does not mention NFS neither. Is there an updated version of the documentation on how to run NFS over RDMA? I have noticed, that it is part of the vanilla Linux kernel but I still did not manage to get it up and running. Many thanks for your advice, Philip -- Philip Frey IBM Zurich Research Laboratory Saumerstrasse 4 | Phone: +41 44 724 8613 CH-8803 Rueschlikon/Switzerland | Email: phf at zurich.ibm.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From YJia at tmriusa.com Wed Mar 4 09:08:49 2009 From: YJia at tmriusa.com (Yicheng Jia) Date: Wed, 4 Mar 2009 11:08:49 -0600 Subject: [ofa-general] link width problem of Qlogic 9024 unmanaged switch Message-ID: Hi, I am running into a performance issue while I am doing OpenSM restart test with Qlogic 9024 24-ports unmanaged switch. In my test environment, 5 nodes with Mellanox HCAs directly connect with the switch. During each test iteration, all nodes reboot and then OpenSM starts to initiate the subnet. The problem I experience is that some time during OpenSM initiates, some ports are reporting 1x link width (I turn on debug option of OpenSM), thus the data transfer performance drops down significantly. I believe it's not cable/physics connection issue because the 1x link width port randomly occurs in different ports, and if I use Flextronics unmanaged switch, the problem is gone. I am wondering if anyone experiences the same problem before, any hint of how to fix it? Thanks! Yicheng Jia _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From vuhuong at mellanox.com Wed Mar 4 10:43:52 2009 From: vuhuong at mellanox.com (Vu Pham) Date: Wed, 04 Mar 2009 10:43:52 -0800 Subject: [ofa-general] NFS over RDMA In-Reply-To: References: Message-ID: <49AECBE8.4070707@mellanox.com> The easiest steps to have nfsrdma up and running: Download OFED-1.4.1-rc1 from http://www.openfabrics.org/downloads/OFED/ofed-1.4.1/ and build/install nfsrdma. It is running on 2.6.27/26/25 and RHEL 5.2 default kernels Or + Download kernel 2.6.27 from kernel.org + Build the kernel 2.6.27 with IB driver (and other modules ie. user_mad, addr_trans, ib_mthca, mlx4_ib... ), nfs client/server, sunrpc... as modules + Boot up with new kernel + Follow the attached README NOTE: You can also download and run nfsrdma on latest development kernel 2.6.29-rc7; however, it does not work on Mellanox ConnectX HCA due to a bug on Fast memory registration work request. We are debugging it now -vu > > Hello, > > I want to try NFS over RDMA. According to the documentation enclosed > within the OFED-1.4 release, it should be as simple as setting > "nfsrdma=y". > > Unfortunately there is no such option. The interactive menu offered by > './install.pl' does not mention NFS neither. > > Is there an updated version of the documentation on how to run NFS over > RDMA? I have noticed, that it is part of the vanilla Linux kernel but > I still did > not manage to get it up and running. > > Many thanks for your advice, > Philip > > -- > Philip Frey > IBM Zurich Research Laboratory > Saumerstrasse 4 | Phone: +41 44 > 724 8613 > CH-8803 Rueschlikon/Switzerland | Email: phf at zurich.ibm.com > ------------------------------------------------------------------------ > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: nfs-rdma.txt URL: From cameron at harr.org Wed Mar 4 10:48:34 2009 From: cameron at harr.org (Cameron Harr) Date: Wed, 04 Mar 2009 11:48:34 -0700 Subject: [ofa-general] NFS over RDMA In-Reply-To: <49AECBE8.4070707@mellanox.com> References: <49AECBE8.4070707@mellanox.com> Message-ID: <49AECD02.6090702@harr.org> An HTML attachment was scrubbed... URL: From hal.rosenstock at gmail.com Wed Mar 4 10:50:53 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 4 Mar 2009 13:50:53 -0500 Subject: ***SPAM*** Re: [ofa-general] ***SPAM*** opensm send_trap144 question In-Reply-To: References: Message-ID: On Wed, Mar 4, 2009 at 10:24 AM, Hal Rosenstock wrote: > Sasha, > > In commits 663255db91ecd2baf78b28d77b4397314e5349ca and > e59489bf06bfed7491b7c3da64ca22ed326b1563, there is the potential for > trap 144 to be generated. We've discussed some aspects of this when > the patches were first on the list. There's another aspect of this > that comes to light now that I've started playing with this. In > osm_req.c, osm_send_trap144 does not set (or for that matter currently > clear) the resp_expected flag. It seems to me that a more robust > strategy would be to set resp_expected to try to make sure that the > other side sees the trap by handling the (trap repress) response matched by transaction ID > (and the timeout/retry strategy is used) if the trap repress is not received > There are a number of underpinnings to change to do this. Would you > accept a patch along those lines ? > > -- Hal > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From Jeffrey.C.Becker at nasa.gov Wed Mar 4 10:54:32 2009 From: Jeffrey.C.Becker at nasa.gov (Jeff Becker) Date: Wed, 04 Mar 2009 10:54:32 -0800 Subject: [ofa-general] NFS over RDMA In-Reply-To: <49AECBE8.4070707@mellanox.com> References: <49AECBE8.4070707@mellanox.com> Message-ID: <49AECE68.2020703@nasa.gov> Hi. If you use install.pl, I think you need to select custom install (not "install all"). This will allow you to pick the nfsrdma option. -jeff Vu Pham wrote: > The easiest steps to have nfsrdma up and running: > Download OFED-1.4.1-rc1 from > http://www.openfabrics.org/downloads/OFED/ofed-1.4.1/ > and build/install nfsrdma. It is running on 2.6.27/26/25 and RHEL 5.2 > default kernels > > Or > + Download kernel 2.6.27 from kernel.org > + Build the kernel 2.6.27 with IB driver (and other modules ie. > user_mad, addr_trans, ib_mthca, mlx4_ib... ), nfs client/server, > sunrpc... as modules > + Boot up with new kernel > + Follow the attached README > > NOTE: You can also download and run nfsrdma on latest development kernel > 2.6.29-rc7; however, it does not work on Mellanox ConnectX HCA due to a > bug on Fast memory registration work request. We are debugging it now > > -vu > >> Hello, >> >> I want to try NFS over RDMA. According to the documentation enclosed >> within the OFED-1.4 release, it should be as simple as setting >> "nfsrdma=y". >> >> Unfortunately there is no such option. The interactive menu offered by >> './install.pl' does not mention NFS neither. >> >> Is there an updated version of the documentation on how to run NFS over >> RDMA? I have noticed, that it is part of the vanilla Linux kernel but >> I still did >> not manage to get it up and running. >> >> Many thanks for your advice, >> Philip >> >> -- >> Philip Frey >> IBM Zurich Research Laboratory >> Saumerstrasse 4 | Phone: +41 44 >> 724 8613 >> CH-8803 Rueschlikon/Switzerland | Email: phf at zurich.ibm.com >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >> > > From dotanba at gmail.com Wed Mar 4 11:25:31 2009 From: dotanba at gmail.com (Dotan Barak) Date: Wed, 04 Mar 2009 21:25:31 +0200 Subject: [ofa-general] puzzled by ATOMIC operation In-Reply-To: <62a06b6fde0e9edc1fef0abc31a89abc.squirrel@webmail.ict.ac.cn> References: <62a06b6fde0e9edc1fef0abc31a89abc.squirrel@webmail.ict.ac.cn> Message-ID: <49AED5AB.5000305@gmail.com> zhouyonghao at ict.ac.cn wrote: > Hi, all > these days I'm puzzled by a weird mistake in atomic operation, when I > use FETCH_AND_ADD, the remote buffer only changed for 9 times at most, > aka if the initial value is 0, when I execute 10 times FETCH_AND_ADD > operations, the value is 9, and no more changes happens to the buffer. > Should I take care of the big-edian from/to litter-edian translation? > No, you should write the value in local host endian. > I tried this, but it seems not the point. > Is there some sample code for atomic operation just like "perftest" in > the libibverbs package? Maybe I missed something? I paste some code > below for reviewing, thanks in advance! > You didn't miss it, not many people use the atomic operations and there isn't any example that uses it. > struct ibv_send_wr wr; > It will be wise to fill the following delaration here: memset(&wr, 0, sizeof(wr)); > wr.opcode = IBV_WR_ATOMIC_FETCH_AND_ADD; > wr.wr.atomic.remote_addr = rem_dest->vaddr; > wr.wr.atomic.rkey = rem_dest->rkey; > wr.send_flags = IBV_SEND_SIGNALED | > IBV_SEND_FENCE; > wr.wr.atomic.compare_add = 1; > wr.wr.atomic.swap = 0; > since you are using Fetch&Add the swap attribute is useless. > wr.wr_id = id; > wr.sg_list = &list; > wr.num_sge = 1; > wr.next = NULL; > What about the value of the s/g entry? > the qp and memory region is set with atomic flag : > mr = ibv_reg_mr(host->pd, buf, len, > IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE > | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_ATOMIC); > > attr.qp_access_flags = IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE > | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_ATOMIC; > At least the QP/MR in the remote side should enable REMOTE_ATOMIC (if you want to enable it locally, it is fine but not really needed). > some attibutes in my ib device about atomic operation are as below: > max_qp_rd_atom is 4, max_qp_init_rd_atom is 128, max_res_rd_atom is 258048 > max_qp_rd_atom: The maximum number of outstanding atomic/RDMA read that a QP can handle as target. max_qp_init_rd_atom: The maximum number of outstanding atomic/RDMA read that a QP can handle as initiator. Obviously, you should use the minimum of the values (use the right value in each side, since different HCAs can have different attributes) I think that max_res_rd_atom is the total outstanding number of atomic/RDMA read that the HCA can handle, but i'm not really sure. If you think that you have a bug, please review the code in remote side (maybe you have problem to read the data...). I hope i helped you a little ... Dotan From devel at morey-chaisemartin.com Wed Mar 4 12:26:51 2009 From: devel at morey-chaisemartin.com (Nicolas Morey-Chaisemartin) Date: Wed, 04 Mar 2009 21:26:51 +0100 Subject: [ofa-general] [RFC] Fat-Tree upgrades In-Reply-To: <49AEA036.2090005@dev.mellanox.co.il> References: <498FDDD5.1090204@ext.bull.net> <49AE54BF.3080000@dev.mellanox.co.il> <49AE69F4.2080903@ext.bull.net> <49AEA036.2090005@dev.mellanox.co.il> Message-ID: <49AEE40B.9000101@morey-chaisemartin.com> Yevgeny Kliteynik a écrit : > Hi Nicolas, > > Nicolas Morey Chaisemartin wrote: >> Yevgeny Kliteynik wrote: >>> Nicolas Morey Chaisemartin wrote: >>>> Hi everyone, >>>> >>>> We have been working quite a lot at Bull lately on the Ftree algorithm >>>> and we have made some upgrades. >>>> However, as they modify the behavior of the ftree algorithm, we >>>> haven't pushed them until now. >>>> I'm just going to detail which upgrades we have done and let you >>>> decide if you are interested, if and how they should be pushed >>>> upstream (new routing algorithm, option in the ftree, etc.) >>>> >>>> Here is a simplify model of the topology we have been working on >>>> >>>> L3 L3 >>>> ___________________|__|____________________ >>>> / / \ \ <= >>>> All the L2 are connected on 2 L3 switches >>>> L2-1 L2-2 L2-1 L2-2 <= >>>> There are service nodes connected directly on L2 switches >>>> / \S1 / \S2 S3/ \ S4/ \ <== >>>> The Nth L1 of a set leads only to the Nth L2 (L2-N). With some >>>> pruning. >>>> L1 L1 L1 L1 >>>> /|\ /|\ /|\ /|\ >>>> ==Fully mixed to L1== ==Fully mixed to L1== <=== We have >>>> multiple set. In each set, all L0 lead to all L1 of their set. >>>> >>>> L0 L0 L0 L0 >>>> / \ / \ / \ / \ >>>> CN CN .. CN CN .... CN CN .. CN CN >>>> >>>> To detail: >>>> We have a bunch of sets. Each set contains compute node, L0 and L1 >>>> switches. >>>> Plus a common top of L2 and L3 switches. >>>> >>>> In each set, there are groups of compute nodes. Each group is >>>> connected to a single L0 switch. >>>> In a given set, all L0 are connected to all L1. >>>> >>>> The Nth L1 of a set is connected to the Nth L2 and only to this one. >>>> (so through a L2, the Nth L1 can only see the Nth L1 of the other sets) >>>> There are Services nodes connected to the L2 switches. >>>> All the L2 are connected to a couple of L3. >>>> >>>> The problem we have seen when routing on this topology is that most of >>>> the routes from CN to SN (service nodes) go through the L3 switches. >>>> With the current algorithm, the less loaded link is choosed to go down >>>> by going up. Therefore, the primary path goes through a L2, then a L3 >>>> from where it covers all the network. >>> Not sure I understand why. >>> There are two stages of routing: >>> 1. Route the downward paths, which is done by climbing up the tree. >>> 2. Route the upward paths, which is done by descending. >>> Before each climbing, the routing first configures all the upward >>> paths: the first acll in the beginning of the >>> __osm_ftree_fabric_route_downgoing_by_going_up() function is >>> __osm_ftree_fabric_route_upgoing_by_going_down(). >> >> OK >> >>> So in your example, if some Service Node (SN) is connected to L2-i >>> switch, the routing should first configure all the i'th switches >>> in the L1 sets, and then all the L0 switches in all the sets because >>> L1 is fully mixed to L0 in each set. >>> >>> Where am I wrong? >> >> You're right. Routes towards the service nodes are OK. >> >> Problem is the routes from the service node. >> >> Let's take a CN (called C0) attached to a L0. >> Algorithm go one step up (downward route) to L1-1 (could be any L1 in >> its set), covers all the upward routes so we have all the routes >> CN->C0 where the CN belong to the same set. >> Next step: Algorithm doesn't have a choice here, it goes to L2-1, >> covers the upward routes so we have all the CN->C0 routes and S1->C0 >> However we don't have any routes from S[234]->C0 yet. >> Next and final step, algorithm go to either one of the L3, covers >> upwards routes and create routes from S[234]->C0. These routes have >> more hops than the minimum possible. > > OK, I get it now. > So there's is a general issue with the existing algorithm: > in topologies with redundant switch level(s), such as L3 in > your example, fat-tree might create switch-to-CN routes (or > in your example, SN-to-CN when SN is on a higher tree level > than CN) that are longer than min-hop. > > Currently, when the algorithm goes up, it checks whether or > not the upper switch already has this LID in the LFT, and > if it has - the route is abandoned. > > What you're suggesting is to check the path length, and if > the new path is shorter, then it should replace the old one. > Sounds like a great idea. > Thanks >>>> This wasn't acceptable for us as L3 switches would be overloaded when >>>> there were less loaded/shorter paths to achieve the same HCA. >>>> >>>> So what we have introduced here is a "balanced min_hop" within the >>>> ftree algorithm. >>>> Basically, instead of just leaving when we reach a LFT which has >>>> already been configured for the target lid, we check the hops count of >>>> the switch toward this lid, and the hop count on the path we came >>>> through. If we have found a shorter path, we update the LFT and minhop >>>> tables to use this new path. > > And what do you do after that? Do you continue climbing > the tree and checking the route length? I guess you need > to continue climbing till all the upper switches will > have LFT entries with route length equal to min-hop. It depends on what you call climbing. On its way up (creating downward routes) you can't find a shorter path (as long as a switch only belongs to one level). On your way down (upward routes) you have to carry to propagate the new shorter path. > I do have some concerns about the CN-to-CN routes in this > case. Any chance these routes might be affected? > > My intuition says that they won't, because they are required > to reside at the same tree level, and since there is full > connectivity between them, they will be all configured with > the shortest path. But I'm not done convincing myself... > Yes. On irregular topology (switch/link loss) you may see such a thing. If we take our example again. Imagine the link between L2-1 and the first L1 of the second set broke down. We take the same way I describe earlier (C0-->L1-1->L2-1) As your link broke down you can't reach the second set from here so the main route will go through L3. However using the solution I proposed earlier, another route will be created through another L2. But imho, it's a good thing. The important idea coming with this fix is that there isn't much difference anymore between primary path and secondary paths. A primary path is just the first path to be explore while the secondary is a later one. On regular topology, the primary path will always be used. However, as soon as your topology gets irregular (which on big networks due to link/switch problems may be quite often), the effects of the minhop start kicking in, and according to our tests we get _much better_ results. To achieve this, more than testing if your path is shorter, you need to equilibrate the secondary routes so they are exlored in the right order. If your first secondary route is a really loaded one, the minhop may create a new route going through here which is a bad thing. What I have done to fix this is instead of finding the lowest loaded port group (primary) and then explore the others (secondaries), I sort them all and explore them in sorted order. To test these algorithms we have written some scripts. We basically calculate the number of routes (CN to CN) per link on the whole network. The better the algorithm is, the less point you should have (on regular topology, theoretical points is about one per level). We mesure this for each switch/link loss using ibsim/opensm and aggregate all the results. Using the minhop and sorted secondary routes, the behaviour of the algorithm is totally changed, in well :) I'm also working on some more enhancements and speed upgrades (sorting costs a bit of time but I've managed to reduce the cost a lot). I'll present them when they are ready. Anyway if you feel like this idea could be integrated in the ftree algorithm, I have patches working, I'll just need to repack them nicely and send them. Nicolas From randy.halverson at qlogic.com Wed Mar 4 14:29:13 2009 From: randy.halverson at qlogic.com (Randy Halverson) Date: Wed, 4 Mar 2009 16:29:13 -0600 Subject: [ofa-general] link width problem of Qlogic 9024 unmanaged switch Message-ID: <88EC963376E93B4DB0F2A69D932F786903D89C9B@MNEXMB2.qlogic.org> Hello Yicheng, Because this is an un-managed switch, not much can be done here unless you have the QLogic FastFabric tools to externally manage the switch; like updating to a newer firmware. Can you provide the s/n# of the switch? It's possible that you are running older firmware on it. How old is this switch? Randy Technical Support QLogic Corporation -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdreier at cisco.com Wed Mar 4 15:23:48 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 04 Mar 2009 15:23:48 -0800 Subject: [ofa-general] [PATCH] IB: Remove useless ibdev_is_alive() tests from sysfs code Message-ID: Some attribute show functions test ibdev_is_alive() to make sure that it's OK to access device state. However, the sysfs attributes will not be registered until the device is fully initialized, and they'll be unregistered before anything is torn down, so ibdev_is_alive() doesn't do anything useful. Remove it. Signed-off-by: Roland Dreier --- The sysfs fix seems to make this cleanup valid now. drivers/infiniband/core/sysfs.c | 16 ---------------- 1 files changed, 0 insertions(+), 16 deletions(-) diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c index 5270aeb..5c04cfb 100644 --- a/drivers/infiniband/core/sysfs.c +++ b/drivers/infiniband/core/sysfs.c @@ -66,11 +66,6 @@ struct port_table_attribute { int index; }; -static inline int ibdev_is_alive(const struct ib_device *dev) -{ - return dev->reg_state == IB_DEV_REGISTERED; -} - static ssize_t port_attr_show(struct kobject *kobj, struct attribute *attr, char *buf) { @@ -80,8 +75,6 @@ static ssize_t port_attr_show(struct kobject *kobj, if (!port_attr->show) return -EIO; - if (!ibdev_is_alive(p->ibdev)) - return -ENODEV; return port_attr->show(p, port_attr, buf); } @@ -562,9 +555,6 @@ static ssize_t show_node_type(struct device *device, { struct ib_device *dev = container_of(device, struct ib_device, dev); - if (!ibdev_is_alive(dev)) - return -ENODEV; - switch (dev->node_type) { case RDMA_NODE_IB_CA: return sprintf(buf, "%d: CA\n", dev->node_type); case RDMA_NODE_RNIC: return sprintf(buf, "%d: RNIC\n", dev->node_type); @@ -581,9 +571,6 @@ static ssize_t show_sys_image_guid(struct device *device, struct ib_device_attr attr; ssize_t ret; - if (!ibdev_is_alive(dev)) - return -ENODEV; - ret = ib_query_device(dev, &attr); if (ret) return ret; @@ -600,9 +587,6 @@ static ssize_t show_node_guid(struct device *device, { struct ib_device *dev = container_of(device, struct ib_device, dev); - if (!ibdev_is_alive(dev)) - return -ENODEV; - return sprintf(buf, "%04x:%04x:%04x:%04x\n", be16_to_cpu(((__be16 *) &dev->node_guid)[0]), be16_to_cpu(((__be16 *) &dev->node_guid)[1]), -- 1.6.0.4 From rdreier at cisco.com Wed Mar 4 15:32:43 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 04 Mar 2009 15:32:43 -0800 Subject: [ofa-general][PATCH 1/2] mlx4_core: Link sensing support In-Reply-To: <49AD25CE.8010208@mellanox.co.il> (Yevgeny Petrilin's message of "Tue, 03 Mar 2009 14:42:54 +0200") References: <49AD25CE.8010208@mellanox.co.il> Message-ID: > mlx4_core-y := alloc.o catas.o cmd.o cq.o eq.o fw.o icm.o intf.o main.o mcg.o \ > - mr.o pd.o port.o profile.o qp.o reset.o srq.o > + mr.o pd.o port.o profile.o qp.o reset.o srq.o sense.o trivial but if this is going to be respun, please keep the filenames in alphabetical order -- ie sense.o goes before srq.o > +#include this isn't needed, is it? > + if (sense->resched) Do we need resched? Can't we just use cancel_delayed_work_sync() when we want to stop this from running? > + queue_delayed_work(sense->sense_wq , &sense->sense_poll, > + round_jiffies(MLX4_SENSE_RANGE)); should be round_jiffies_relative, right? > +void mlx4_stop_sense(struct mlx4_dev *dev) > +{ > + mlx4_priv(dev)->sense.resched = 0; > +} This doesn't stop anything... we need cancel_delayed_work_sync(). > + sense->sense_wq = create_singlethread_workqueue("mlx4_sense"); Do we really another work queue, or can we share one queue for the catastrophic error and port sensing work? - R. From rdreier at cisco.com Wed Mar 4 15:33:27 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 04 Mar 2009 15:33:27 -0800 Subject: [ofa-general][PATCH 2/2] mlx4_core: Don't perform SET_PORT command for Eth port from core. In-Reply-To: <49AD25DE.9010501@mellanox.co.il> (Yevgeny Petrilin's message of "Tue, 03 Mar 2009 14:43:10 +0200") References: <49AD25DE.9010501@mellanox.co.il> Message-ID: I don't really follow this. Is this fixing an existing bug, or is it something that gets broken if patch 1/2 is applied? - R. From Jie.Cai at cs.anu.edu.au Wed Mar 4 16:40:01 2009 From: Jie.Cai at cs.anu.edu.au (Jie Cai) Date: Thu, 05 Mar 2009 11:40:01 +1100 Subject: [ofa-general] Bandwidth of performance with multirail IB In-Reply-To: References: <20090223211155.730AFE28137@openfabrics.org> <49A378BC.5010806@cs.anu.edu.au> <200902240941.58634.cap@nsc.liu.se> <49A7885B.3010005@cs.anu.edu.au> Message-ID: <49AF1F61.1090908@cs.anu.edu.au> Davis, Arlin R wrote: > > Jie, > > >> A question on implementation multi-rail with uDAPL connections. >> >> What I did is open 2 IAs (corresponding to the 2 ports on >> HCAs) on each >> node. >> Then create one EP for each IA, and connect those EPs to the >> corresponding >> EP at other node. >> >> Then data been transferred via both EP-connections. >> > > Did you verify data flow is across independent HCA channels > using perfquery? > I have using perfquery to verified that data has been transmit through both channel. The ourput for perfquery is: ############# Before data transfer: ############# # Port counters: Lid 8 port 1 PortSelect:......................1 CounterSelect:...................0x0000 LinkDowned:......................2 XmtDiscards:.....................39 XmtData:.........................4294967295 RcvData:.........................20456971 XmtPkts:.........................112151685 RcvPkts:.........................2372783 # Port counters: Lid 9 port 2 PortSelect:......................2 CounterSelect:...................0x0000 LinkDowned:......................1 XmtDiscards:.....................21 XmtData:.........................4294967295 RcvData:.........................7123011 XmtPkts:.........................62927413 RcvPkts:.........................896106 ############# After data transfer: ############# # Port counters: Lid 8 port 1 PortSelect:......................1 CounterSelect:...................0x0000 LinkDowned:......................2 XmtDiscards:.....................39 XmtData:.........................4294967295 RcvData:.........................20457865 XmtPkts:.........................112168095 RcvPkts:.........................2372882 # Port counters: Lid 9 port 2 PortSelect:......................2 CounterSelect:...................0x0000 LinkDowned:......................1 XmtDiscards:.....................21 XmtData:.........................4294967295 RcvData:.........................7123912 XmtPkts:.........................62943824 RcvPkts:.........................896206 Difference in XmtPkts of port1 is 16410, and the difference for port 2 is 16411. Does this suggest that around similar size of data has been transfered through both rails? > >> I have been notice that there's a MULTIPATH connection flag for dapl, >> but I did >> not use it. What's the use of it? >> > > Multipath is not supported at this time. > > I will roll up a uDAPL test and see if I can verify performance. > > -arlin > > From randy.halverson at qlogic.com Wed Mar 4 16:39:54 2009 From: randy.halverson at qlogic.com (Randy Halverson) Date: Wed, 4 Mar 2009 18:39:54 -0600 Subject: [ofa-general] link width problem of Qlogic 9024 unmanaged switch Message-ID: <88EC963376E93B4DB0F2A69D932F786903D89CAF@MNEXMB2.qlogic.org> Hello Yicheng, After checking internally, this appears to be a known problem with older firmware for the 9024FC switches. It appears that you or another person at 'tmriusa.com' has recently opened a case with QLogic Tech Support for this issue. Please continue to work with QLogic Tech Support on firmware upgrade resolution since you probably don't have our FastFabric Tools to manage the 9024FC switches.. Regards, Randy Technical Support QLogic Corporation -------------- next part -------------- An HTML attachment was scrubbed... URL: From poknam at gmail.com Wed Mar 4 21:51:28 2009 From: poknam at gmail.com (PN) Date: Thu, 5 Mar 2009 13:51:28 +0800 Subject: ***SPAM*** Re: [ofa-general] OFED 1.4.1 is available In-Reply-To: <5D49E7A8952DC44FB38C38FA0D758EAD01F50B50@mtlexch01.mtl.com> References: <5D49E7A8952DC44FB38C38FA0D758EAD01F50B50@mtlexch01.mtl.com> Message-ID: <92daa7bf0903042151l52d77c20r2440d95c3be49de8@mail.gmail.com> HI, I have a build error of OFED-1.4.1-rc1 under CentOS 5.2: ..... Build openmpi_gcc RPM Running rpmbuild --rebuild --define '_topdir /var/tmp/OFED_topdir' --define 'dist %{nil}' --target x86_64 --define '_name openmpi_gcc' --define 'mpi_selector /usr/bin/mpi-selector' --define 'use_mpi_selector 1' --define 'install_shell_scripts 1' --define 'shell_scripts_basename mpivars' --define '_usr /usr' --define 'ofed 0' --define '_prefix /usr/mpi/gcc/openmpi-1.3.1rc4' --define '_defaultdocdir /usr/mpi/gcc/openmpi-1.3.1rc4' --define '_mandir %{_prefix}/share/man' --define 'mflags -j 4' --define 'configure_options --with-openib=/usr --with-openib-libdir=/usr/lib64 CC=gcc CXX=g++ F77=gfortran FC=gfortran --enable-mpirun-prefix-by-default' --define 'use_default_rpm_opt_flags 1' /opt/software/packages/ofed/OFED-1.4.1-rc1/OFED-1.4.1-rc1/SRPMS/openmpi-1.3.1rc4-1.src.rpm Failed to build openmpi RPM See /tmp/OFED.28377.logs/openmpi.rpmbuild.log In /tmp/OFED.28377.logs/openmpi.rpmbuild.log: ..... gcc -DHAVE_CONFIG_H -I. -I.. -I../tools/opari/lib -I../extlib/otf/otflib -I../extlib/otf/otflib -D_GNU_SOURCE -DBINDIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/bin\" -DDATADIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/share\" -DRFG -DVT_MEMHOOK -DVT_IOWRAP -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -MT vt_iowrap_helper.o -MD -MP -MF .deps/vt_iowrap_helper.Tpo -c -o vt_iowrap_helper.o vt_iowrap_helper.c mv -f .deps/vt_memhook.Tpo .deps/vt_memhook.Po gcc -DHAVE_CONFIG_H -I. -I.. -I../tools/opari/lib -I../extlib/otf/otflib -I../extlib/otf/otflib -D_GNU_SOURCE -DBINDIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/bin\" -DDATADIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/share\" -DRFG -DVT_MEMHOOK -DVT_IOWRAP -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -MT rfg_regions.o -MD -MP -MF .deps/rfg_regions.Tpo -c -o rfg_regions.o rfg_regions.c vt_iowrap.c:1242: error: expected declaration specifiers or '...' before numeric constant vt_iowrap.c:1243: error: conflicting types for '__fprintf_chk' mv -f .deps/vt_iowrap_helper.Tpo .deps/vt_iowrap_helper.Po make[5]: *** [vt_iowrap.o] Error 1 make[5]: *** Waiting for unfinished jobs.... mv -f .deps/vt_comp_gnu.Tpo .deps/vt_comp_gnu.Po mv -f .deps/rfg_regions.Tpo .deps/rfg_regions.Po make[5]: Leaving directory `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt/vt/vtlib' make[4]: *** [all-recursive] Error 1 make[4]: Leaving directory `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt/vt' make[3]: *** [all] Error 2 make[3]: Leaving directory `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt/vt' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi' make: *** [all-recursive] Error 1 error: Bad exit status from /var/tmp/rpm-tmp.40739 (%build) RPM build errors: user jsquyres does not exist - using root group eng10 does not exist - using root user jsquyres does not exist - using root group eng10 does not exist - using root Bad exit status from /var/tmp/rpm-tmp.40739 (%build) The error seems similar to http://www.open-mpi.org/community/lists/devel/2009/01/5230.php Regards, PN 2009/3/5 Tziporet Koren > Hi, > > OFED-1.4.1-rc1 release is available on > > *http://www.openfabrics.org/downloads/OFED/ofed-1.4.1/OFED-1.4.1-rc1.tgz* > > To get BUILD_ID run ofed_info > > Please report any issues in bugzilla *https://bugs.openfabrics.org/*for OFED 1.4.1 > > Vladimir & Tziporet > > ======================================================================== > > Release information: > > ------------------------------ > > Linux Operating Systems: > > - RedHat EL4 up4: 2.6.9-42.ELsmp * > > - RedHat EL4 up5: 2.6.9-55.ELsmp > > - RedHat EL4 up6: 2.6.9-67.ELsmp > > - RedHat EL4 up7: 2.6.9-78.ELsmp > > - RedHat EL5: 2.6.18-8.el5 > > - RedHat EL5 up1: 2.6.18-53.el5 > > - RedHat EL5 up2: 2.6.18-92.el5 > > - RedHat EL5 up3: 2.6.18-128.el5 > > - OEL 4.5: 2.6.9-55.ELsmp > > - OEL 5.2: 2.6.18-92.el5 > > - CentOS 5.2: 2.6.18-92.el5 > > - Fedora C9: 2.6.25-14.fc9 * > > - SLES10: 2.6.16.21-0.8-smp > > - SLES10 SP1: 2.6.16.46-0.12-smp > > - SLES10 SP1 up1: 2.6.16.53-0.16-smp > > - SLES10 SP2: 2.6.16.60-0.21-smp > > - SLES11 RC3: 2.6.27.13-1-default > > - OpenSuSE 10.3: 2.6.22.5-31 * > > - kernel.org: 2.6.26 and 2.6.27 > > * Minimal QA for these versions > > Systems: > > * x86_64 > > * x86 > > * ia64 > > * ppc64 > > Main Changes from OFED-1.4 > > ====================== > > - Added support for RHEL 5.3 and SLES11 RC3 > > - Added RHEL 5.2 backports for NFS/RDMA > > - Updated MPI packages: > > mvapich-1.1.0-3143 > > Open MPI 1.3.1-rc4 > > - Updated bonding package: ib-bonding-0.9.0-38 > > - Updated opensm version to include critical bug fixes > > - Fixed RDS iWARP support > > - Low level drivers updated: ehca, mlx4, cxgb3, nes > > - Bug fixes > > - Attached kernel git tree changes for details > > <> > > Tasks that should be completed for RC2: > > ============================= > > 1. High priority bug fixes > > 2. RHEL 5.2 32 bits compilation issue (due to NFS/RDMA backports) > > 3. Open MPI 1.3.1 release > > 4. MVAPICH 1.1.1.minor release with critical bug fixes > > 5. OSM release 3.2.6 > > 6. Documentation update > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > -------------- next part -------------- An HTML attachment was scrubbed... URL: From viral.vkm at gmail.com Wed Mar 4 22:50:16 2009 From: viral.vkm at gmail.com (vkm) Date: Thu, 05 Mar 2009 12:20:16 +0530 Subject: ***SPAM*** Re: [ofa-general] OFED 1.4.1 is available In-Reply-To: <49AF7265.4020506@gmail.com> References: <5D49E7A8952DC44FB38C38FA0D758EAD01F50B50@mtlexch01.mtl.com> <92daa7bf0903042151l52d77c20r2440d95c3be49de8@mail.gmail.com> <49AF6D11.70600@gmail.com> <49AF7265.4020506@gmail.com> Message-ID: <49AF7628.5010905@gmail.com> Hi, JFYI, A workaround mentioned here http://www.open-mpi.org/community/lists/devel/2009/01/5176.php '--with-contrib-vt-flags=--disable-iotrace' helps. no idea if workaround causes any harm in any way or how good is it to apply !! ?? >> >> PN wrote: >>> HI, >>> >>> I have a build error of OFED-1.4.1-rc1 under CentOS 5.2: >>> >>> ..... >>> Build openmpi_gcc RPM >>> Running rpmbuild --rebuild --define '_topdir /var/tmp/OFED_topdir' >>> --define 'dist %{nil}' --target x86_64 --define '_name openmpi_gcc' >>> --define 'mpi_selector /usr/bin/mpi-selector' --define >>> 'use_mpi_selector 1' --define 'install_shell_scripts 1' --define >>> 'shell_scripts_basename mpivars' --define '_usr /usr' --define 'ofed >>> 0' --define '_prefix /usr/mpi/gcc/openmpi-1.3.1rc4' --define >>> '_defaultdocdir /usr/mpi/gcc/openmpi-1.3.1rc4' --define '_mandir >>> %{_prefix}/share/man' --define 'mflags -j 4' --define >>> 'configure_options --with-openib=/usr >>> --with-openib-libdir=/usr/lib64 CC=gcc CXX=g++ F77=gfortran >>> FC=gfortran --enable-mpirun-prefix-by-default' --define >>> 'use_default_rpm_opt_flags 1' >>> /opt/software/packages/ofed/OFED-1.4.1-rc1/OFED-1.4.1-rc1/SRPMS/openmpi-1.3.1rc4-1.src.rpm >>> >>> Failed to build openmpi RPM >>> See /tmp/OFED.28377.logs/openmpi.rpmbuild.log >>> >>> >>> In /tmp/OFED.28377.logs/openmpi.rpmbuild.log: >>> ..... >>> gcc -DHAVE_CONFIG_H -I. -I.. -I../tools/opari/lib >>> -I../extlib/otf/otflib -I../extlib/otf/otflib -D_GNU_SOURCE >>> -DBINDIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/bin\" >>> -DDATADIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/share\" -DRFG >>> -DVT_MEMHOOK -DVT_IOWRAP -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 >>> -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 >>> -mtune=generic -MT vt_iowrap_helper.o -MD -MP -MF >>> .deps/vt_iowrap_helper.Tpo -c -o vt_iowrap_helper.o vt_iowrap_helper.c >>> mv -f .deps/vt_memhook.Tpo .deps/vt_memhook.Po >>> gcc -DHAVE_CONFIG_H -I. -I.. -I../tools/opari/lib >>> -I../extlib/otf/otflib -I../extlib/otf/otflib -D_GNU_SOURCE >>> -DBINDIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/bin\" >>> -DDATADIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/share\" -DRFG >>> -DVT_MEMHOOK -DVT_IOWRAP -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 >>> -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 >>> -mtune=generic -MT rfg_regions.o -MD -MP -MF .deps/rfg_regions.Tpo >>> -c -o rfg_regions.o rfg_regions.c >>> vt_iowrap.c:1242: error: expected declaration specifiers or '...' >>> before numeric constant >>> vt_iowrap.c:1243: error: conflicting types for '__fprintf_chk' >>> mv -f .deps/vt_iowrap_helper.Tpo .deps/vt_iowrap_helper.Po >>> make[5]: *** [vt_iowrap.o] Error 1 >>> make[5]: *** Waiting for unfinished jobs.... >>> mv -f .deps/vt_comp_gnu.Tpo .deps/vt_comp_gnu.Po >>> mv -f .deps/rfg_regions.Tpo .deps/rfg_regions.Po >>> make[5]: Leaving directory >>> `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt/vt/vtlib' >>> make[4]: *** [all-recursive] Error 1 >>> make[4]: Leaving directory >>> `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt/vt' >>> make[3]: *** [all] Error 2 >>> make[3]: Leaving directory >>> `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt/vt' >>> make[2]: *** [all-recursive] Error 1 >>> make[2]: Leaving directory >>> `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt' >>> make[1]: *** [all-recursive] Error 1 >>> make[1]: Leaving directory >>> `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi' >>> make: *** [all-recursive] Error 1 >>> error: Bad exit status from /var/tmp/rpm-tmp.40739 (%build) >>> >>> >>> RPM build errors: >>> user jsquyres does not exist - using root >>> group eng10 does not exist - using root >>> user jsquyres does not exist - using root >>> group eng10 does not exist - using root >>> Bad exit status from /var/tmp/rpm-tmp.40739 (%build) >>> >>> >>> The error seems similar to >>> http://www.open-mpi.org/community/lists/devel/2009/01/5230.php >>> >>> Regards, >>> PN >>> >>> >>> 2009/3/5 Tziporet Koren >> > >>> >>> Hi, >>> >>> OFED-1.4.1-rc1 release is available on >>> >>> >>> _http://www.openfabrics.org/downloads/OFED/ofed-1.4.1/OFED-1.4.1-rc1.tgz_ >>> >>> >>> >>> To get BUILD_ID run ofed_info >>> >>> Please report any issues in bugzilla >>> _https://bugs.openfabrics.org/_ for OFED 1.4.1 >>> >>> Vladimir & Tziporet >>> >>> >>> ======================================================================== >>> >>> >>> >>> Release information: >>> >>> ------------------------------ >>> >>> Linux Operating Systems: >>> >>> - RedHat EL4 up4: 2.6.9-42.ELsmp * >>> >>> - RedHat EL4 up5: 2.6.9-55.ELsmp >>> >>> - RedHat EL4 up6: 2.6.9-67.ELsmp >>> >>> - RedHat EL4 up7: 2.6.9-78.ELsmp >>> >>> - RedHat EL5: 2.6.18-8.el5 >>> >>> - RedHat EL5 up1: 2.6.18-53.el5 >>> >>> - RedHat EL5 up2: 2.6.18-92.el5 >>> >>> - RedHat EL5 up3: 2.6.18-128.el5 >>> >>> - OEL 4.5: 2.6.9-55.ELsmp >>> >>> - OEL 5.2: 2.6.18-92.el5 >>> >>> - CentOS 5.2: 2.6.18-92.el5 >>> >>> - Fedora C9: 2.6.25-14.fc9 * >>> >>> - SLES10: 2.6.16.21-0.8-smp >>> >>> - SLES10 SP1: 2.6.16.46-0.12-smp >>> >>> - SLES10 SP1 up1: 2.6.16.53-0.16-smp >>> >>> - SLES10 SP2: 2.6.16.60-0.21-smp >>> >>> - SLES11 RC3: 2.6.27.13-1-default >>> >>> - OpenSuSE 10.3: 2.6.22.5-31 * >>> >>> - kernel.org : 2.6.26 and >>> 2.6.27 >>> >>> * Minimal QA for these versions >>> >>> Systems: >>> >>> * x86_64 >>> >>> * x86 >>> >>> * ia64 >>> >>> * ppc64 >>> >>> >>> Main Changes from OFED-1.4 >>> >>> ====================== >>> >>> - Added support for RHEL 5.3 and SLES11 RC3 >>> >>> - Added RHEL 5.2 backports for NFS/RDMA >>> >>> - Updated MPI packages: >>> >>> mvapich-1.1.0-3143 >>> >>> Open MPI 1.3.1-rc4 >>> >>> - Updated bonding package: ib-bonding-0.9.0-38 >>> >>> - Updated opensm version to include critical bug fixes >>> >>> - Fixed RDS iWARP support >>> >>> - Low level drivers updated: ehca, mlx4, cxgb3, nes >>> >>> - Bug fixes >>> >>> - Attached kernel git tree changes for details >>> >>> <> >>> >>> Tasks that should be completed for RC2: >>> >>> ============================= >>> >>> 1. High priority bug fixes >>> >>> 2. RHEL 5.2 32 bits compilation issue (due to NFS/RDMA backports) >>> >>> 3. Open MPI 1.3.1 release >>> >>> 4. MVAPICH 1.1.1.minor release with critical bug fixes >>> >>> 5. OSM release 3.2.6 >>> >>> 6. Documentation update >>> >>> >>> _______________________________________________ >>> general mailing list >>> general at lists.openfabrics.org >>> >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>> >>> To unsubscribe, please visit >>> http://openib.org/mailman/listinfo/openib-general >>> >>> >>> >>> >>> *Email Scanned for Virus & Dangerous Content by :* >>> *www.CleanMailGateway.com* >>> >>> ------------------------------------------------------------------------ >>> >>> >>> _______________________________________________ >>> general mailing list >>> general at lists.openfabrics.org >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>> >>> To unsubscribe, please visit >>> http://openib.org/mailman/listinfo/openib-general > From poknam at gmail.com Thu Mar 5 00:09:30 2009 From: poknam at gmail.com (PN) Date: Thu, 5 Mar 2009 16:09:30 +0800 Subject: ***SPAM*** Re: [ofa-general] OFED 1.4.1 is available In-Reply-To: <49AF7628.5010905@gmail.com> References: <5D49E7A8952DC44FB38C38FA0D758EAD01F50B50@mtlexch01.mtl.com> <92daa7bf0903042151l52d77c20r2440d95c3be49de8@mail.gmail.com> <49AF6D11.70600@gmail.com> <49AF7265.4020506@gmail.com> <49AF7628.5010905@gmail.com> Message-ID: <92daa7bf0903050009t3ce5a969kbe41028164f38ca6@mail.gmail.com> I can successfully build the rpm of openmpi by adding the flag. Thanks. Regards, PN 2009/3/5 vkm > Hi, > > JFYI, > A workaround mentioned here > http://www.open-mpi.org/community/lists/devel/2009/01/5176.php > '--with-contrib-vt-flags=--disable-iotrace' > helps. > > no idea if workaround causes any harm in any way or how good is it to > apply !! ?? > > > > >>> PN wrote: >>> >>>> HI, >>>> >>>> I have a build error of OFED-1.4.1-rc1 under CentOS 5.2: >>>> >>>> ..... >>>> Build openmpi_gcc RPM >>>> Running rpmbuild --rebuild --define '_topdir /var/tmp/OFED_topdir' >>>> --define 'dist %{nil}' --target x86_64 --define '_name openmpi_gcc' --define >>>> 'mpi_selector /usr/bin/mpi-selector' --define 'use_mpi_selector 1' --define >>>> 'install_shell_scripts 1' --define 'shell_scripts_basename mpivars' --define >>>> '_usr /usr' --define 'ofed 0' --define '_prefix >>>> /usr/mpi/gcc/openmpi-1.3.1rc4' --define '_defaultdocdir >>>> /usr/mpi/gcc/openmpi-1.3.1rc4' --define '_mandir %{_prefix}/share/man' >>>> --define 'mflags -j 4' --define 'configure_options --with-openib=/usr >>>> --with-openib-libdir=/usr/lib64 CC=gcc CXX=g++ F77=gfortran FC=gfortran >>>> --enable-mpirun-prefix-by-default' --define 'use_default_rpm_opt_flags 1' >>>> /opt/software/packages/ofed/OFED-1.4.1-rc1/OFED-1.4.1-rc1/SRPMS/openmpi-1.3.1rc4-1.src.rpm >>>> >>>> Failed to build openmpi RPM >>>> See /tmp/OFED.28377.logs/openmpi.rpmbuild.log >>>> >>>> >>>> In /tmp/OFED.28377.logs/openmpi.rpmbuild.log: >>>> ..... >>>> gcc -DHAVE_CONFIG_H -I. -I.. -I../tools/opari/lib -I../extlib/otf/otflib >>>> -I../extlib/otf/otflib -D_GNU_SOURCE >>>> -DBINDIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/bin\" >>>> -DDATADIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/share\" -DRFG -DVT_MEMHOOK >>>> -DVT_IOWRAP -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions >>>> -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -MT >>>> vt_iowrap_helper.o -MD -MP -MF .deps/vt_iowrap_helper.Tpo -c -o >>>> vt_iowrap_helper.o vt_iowrap_helper.c >>>> mv -f .deps/vt_memhook.Tpo .deps/vt_memhook.Po >>>> gcc -DHAVE_CONFIG_H -I. -I.. -I../tools/opari/lib -I../extlib/otf/otflib >>>> -I../extlib/otf/otflib -D_GNU_SOURCE >>>> -DBINDIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/bin\" >>>> -DDATADIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/share\" -DRFG -DVT_MEMHOOK >>>> -DVT_IOWRAP -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions >>>> -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -MT >>>> rfg_regions.o -MD -MP -MF .deps/rfg_regions.Tpo -c -o rfg_regions.o >>>> rfg_regions.c >>>> vt_iowrap.c:1242: error: expected declaration specifiers or '...' before >>>> numeric constant >>>> vt_iowrap.c:1243: error: conflicting types for '__fprintf_chk' >>>> mv -f .deps/vt_iowrap_helper.Tpo .deps/vt_iowrap_helper.Po >>>> make[5]: *** [vt_iowrap.o] Error 1 >>>> make[5]: *** Waiting for unfinished jobs.... >>>> mv -f .deps/vt_comp_gnu.Tpo .deps/vt_comp_gnu.Po >>>> mv -f .deps/rfg_regions.Tpo .deps/rfg_regions.Po >>>> make[5]: Leaving directory >>>> `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt/vt/vtlib' >>>> make[4]: *** [all-recursive] Error 1 >>>> make[4]: Leaving directory >>>> `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt/vt' >>>> make[3]: *** [all] Error 2 >>>> make[3]: Leaving directory >>>> `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt/vt' >>>> make[2]: *** [all-recursive] Error 1 >>>> make[2]: Leaving directory >>>> `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt' >>>> make[1]: *** [all-recursive] Error 1 >>>> make[1]: Leaving directory >>>> `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi' >>>> make: *** [all-recursive] Error 1 >>>> error: Bad exit status from /var/tmp/rpm-tmp.40739 (%build) >>>> >>>> >>>> RPM build errors: >>>> user jsquyres does not exist - using root >>>> group eng10 does not exist - using root >>>> user jsquyres does not exist - using root >>>> group eng10 does not exist - using root >>>> Bad exit status from /var/tmp/rpm-tmp.40739 (%build) >>>> >>>> >>>> The error seems similar to >>>> http://www.open-mpi.org/community/lists/devel/2009/01/5230.php >>>> >>>> Regards, >>>> PN >>>> >>>> >>>> 2009/3/5 Tziporet Koren >>> tziporet at mellanox.co.il>> >>>> >>>> Hi, >>>> >>>> OFED-1.4.1-rc1 release is available on >>>> >>>> _ >>>> http://www.openfabrics.org/downloads/OFED/ofed-1.4.1/OFED-1.4.1-rc1.tgz_ >>>> >>>> >>>> To get BUILD_ID run ofed_info >>>> >>>> Please report any issues in bugzilla >>>> _https://bugs.openfabrics.org/_ for OFED 1.4.1 >>>> >>>> Vladimir & Tziporet >>>> >>>> >>>> ======================================================================== >>>> >>>> >>>> Release information: >>>> >>>> ------------------------------ >>>> >>>> Linux Operating Systems: >>>> >>>> - RedHat EL4 up4: 2.6.9-42.ELsmp * >>>> >>>> - RedHat EL4 up5: 2.6.9-55.ELsmp >>>> >>>> - RedHat EL4 up6: 2.6.9-67.ELsmp >>>> >>>> - RedHat EL4 up7: 2.6.9-78.ELsmp >>>> >>>> - RedHat EL5: 2.6.18-8.el5 >>>> >>>> - RedHat EL5 up1: 2.6.18-53.el5 >>>> >>>> - RedHat EL5 up2: 2.6.18-92.el5 >>>> >>>> - RedHat EL5 up3: 2.6.18-128.el5 >>>> >>>> - OEL 4.5: 2.6.9-55.ELsmp >>>> >>>> - OEL 5.2: 2.6.18-92.el5 >>>> >>>> - CentOS 5.2: 2.6.18-92.el5 >>>> >>>> - Fedora C9: 2.6.25-14.fc9 * >>>> >>>> - SLES10: 2.6.16.21-0.8-smp >>>> >>>> - SLES10 SP1: 2.6.16.46-0.12-smp >>>> >>>> - SLES10 SP1 up1: 2.6.16.53-0.16-smp >>>> >>>> - SLES10 SP2: 2.6.16.60-0.21-smp >>>> >>>> - SLES11 RC3: 2.6.27.13-1-default >>>> >>>> - OpenSuSE 10.3: 2.6.22.5-31 * >>>> >>>> - kernel.org : 2.6.26 and >>>> 2.6.27 >>>> >>>> * Minimal QA for these versions >>>> >>>> Systems: >>>> >>>> * x86_64 >>>> >>>> * x86 >>>> >>>> * ia64 >>>> >>>> * ppc64 >>>> >>>> >>>> Main Changes from OFED-1.4 >>>> >>>> ====================== >>>> >>>> - Added support for RHEL 5.3 and SLES11 RC3 >>>> >>>> - Added RHEL 5.2 backports for NFS/RDMA >>>> >>>> - Updated MPI packages: >>>> >>>> mvapich-1.1.0-3143 >>>> >>>> Open MPI 1.3.1-rc4 >>>> >>>> - Updated bonding package: ib-bonding-0.9.0-38 >>>> >>>> - Updated opensm version to include critical bug fixes >>>> >>>> - Fixed RDS iWARP support >>>> >>>> - Low level drivers updated: ehca, mlx4, cxgb3, nes >>>> >>>> - Bug fixes >>>> >>>> - Attached kernel git tree changes for details >>>> >>>> <> >>>> >>>> Tasks that should be completed for RC2: >>>> >>>> ============================= >>>> >>>> 1. High priority bug fixes >>>> >>>> 2. RHEL 5.2 32 bits compilation issue (due to NFS/RDMA backports) >>>> >>>> 3. Open MPI 1.3.1 release >>>> >>>> 4. MVAPICH 1.1.1.minor release with critical bug fixes >>>> >>>> 5. OSM release 3.2.6 >>>> >>>> 6. Documentation update >>>> >>>> >>>> _______________________________________________ >>>> general mailing list >>>> general at lists.openfabrics.org >>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>>> >>>> To unsubscribe, please visit >>>> http://openib.org/mailman/listinfo/openib-general >>>> >>>> >>>> >>>> >>>> *Email Scanned for Virus & Dangerous Content by :* * >>>> www.CleanMailGateway.com* >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> >>>> _______________________________________________ >>>> general mailing list >>>> general at lists.openfabrics.org >>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>>> >>>> To unsubscribe, please visit >>>> http://openib.org/mailman/listinfo/openib-general >>>> >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From kliteyn at dev.mellanox.co.il Thu Mar 5 01:04:26 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Thu, 05 Mar 2009 11:04:26 +0200 Subject: [ofa-general] [RFC] Fat-Tree upgrades In-Reply-To: <49AEE40B.9000101@morey-chaisemartin.com> References: <498FDDD5.1090204@ext.bull.net> <49AE54BF.3080000@dev.mellanox.co.il> <49AE69F4.2080903@ext.bull.net> <49AEA036.2090005@dev.mellanox.co.il> <49AEE40B.9000101@morey-chaisemartin.com> Message-ID: <49AF959A.8070907@dev.mellanox.co.il> Hi Nicolas, Nicolas Morey-Chaisemartin wrote: >>>>> So what we have introduced here is a "balanced min_hop" within the >>>>> ftree algorithm. >>>>> Basically, instead of just leaving when we reach a LFT which has >>>>> already been configured for the target lid, we check the hops count of >>>>> the switch toward this lid, and the hop count on the path we came >>>>> through. If we have found a shorter path, we update the LFT and minhop >>>>> tables to use this new path. >> And what do you do after that? Do you continue climbing >> the tree and checking the route length? > I guess you need >> to continue climbing till all the upper switches will >> have LFT entries with route length equal to min-hop. > > > It depends on what you call climbing. > On its way up (creating downward routes) you can't find a shorter path (as long as a switch only belongs to one level). > On your way down (upward routes) you have to carry to propagate the new shorter path. Not exactly. Even on your way up (while creating downward routes) you can find shorter paths than what was configured by main path. It's hard to explain in words, so I draw some ppt slides (see the attachement). >> I do have some concerns about the CN-to-CN routes in this >> case. Any chance these routes might be affected? >> >> My intuition says that they won't, because they are required >> to reside at the same tree level, and since there is full >> connectivity between them, they will be all configured with >> the shortest path. But I'm not done convincing myself... >> > > Yes. On irregular topology (switch/link loss) you may see such a thing. > > If we take our example again. > Imagine the link between L2-1 and the first L1 of the second set broke down. > We take the same way I describe earlier (C0-->L1-1->L2-1) > As your link broke down you can't reach the second set from here so the main route will go through L3. > However using the solution I proposed earlier, another route will be created through another L2. > > But imho, it's a good thing. Agree. > The important idea coming with this fix is that there isn't much difference anymore between primary path and secondary paths. A primary path is just the first path to be explore while the secondary is a later one. > On regular topology, the primary path will always be used. > However, as soon as your topology gets irregular (which on big networks due to link/switch problems may be quite often), the effects of the minhop start kicking in, and according to our tests we get _much better_ results. Great :) > To achieve this, more than testing if your path is shorter, you need to equilibrate the secondary routes so they are exlored in the right order. If your first secondary route is a really loaded one, the minhop may create a new route going through here which is a bad thing. Right, and since the whole point of this min-hop in fat-tree kicks in only in an irregular or faulty topologies, you will have such loaded links by definition. > What I have done to fix this is instead of finding the lowest loaded port group (primary) and then explore the others (secondaries), I sort them all and explore them in sorted order. Sounds great > To test these algorithms we have written some scripts. > We basically calculate the number of routes (CN to CN) per link on the whole network. > The better the algorithm is, the less point you should have (on regular topology, theoretical points is about one per level). > We mesure this for each switch/link loss using ibsim/opensm and aggregate all the results. > Using the minhop and sorted secondary routes, the behaviour of the algorithm is totally changed, in well :) > > I'm also working on some more enhancements and speed upgrades (sorting costs a bit of time but I've managed to reduce the cost a lot). I'll present them when they are ready. The speed penalty was my other concern. Will be happy to see the comparison of the calculation time with and w/o the sorting once its ready. > Anyway if you feel like this idea could be integrated in the ftree algorithm, I have patches working, I'll just need to repack them nicely and send them. Sure, this is a very important addition to the fat-tree routing. -- Yevgeny > Nicolas > -------------- next part -------------- A non-text attachment was scrubbed... Name: fat-tree.ppt Type: application/vnd.ms-powerpoint Size: 72704 bytes Desc: not available URL: From nicolas.morey-chaisemartin at ext.bull.net Thu Mar 5 01:10:44 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Thu, 05 Mar 2009 10:10:44 +0100 Subject: [ofa-general] [RFC] Fat-Tree upgrades In-Reply-To: <49AF959A.8070907@dev.mellanox.co.il> References: <498FDDD5.1090204@ext.bull.net> <49AE54BF.3080000@dev.mellanox.co.il> <49AE69F4.2080903@ext.bull.net> <49AEA036.2090005@dev.mellanox.co.il> <49AEE40B.9000101@morey-chaisemartin.com> <49AF959A.8070907@dev.mellanox.co.il> Message-ID: <49AF9714.2090604@ext.bull.net> Yevgeny Kliteynik wrote: > > Not exactly. Even on your way up (while creating downward routes) > you can find shorter paths than what was configured by main path. > It's hard to explain in words, so I draw some ppt slides (see the > attachement). > You're right. In fact my patches already deal with these cases. I just didn't remember it :) > >> To test these algorithms we have written some scripts. >> We basically calculate the number of routes (CN to CN) per link on the >> whole network. >> The better the algorithm is, the less point you should have (on >> regular topology, theoretical points is about one per level). We >> mesure this for each switch/link loss using ibsim/opensm and aggregate >> all the results. Using the minhop and sorted secondary routes, the >> behaviour of the algorithm is totally changed, in well :) >> >> I'm also working on some more enhancements and speed upgrades (sorting >> costs a bit of time but I've managed to reduce the cost a lot). I'll >> present them when they are ready. > > The speed penalty was my other concern. > Will be happy to see the comparison of the calculation time with and w/o > the sorting once its ready. I'll run some bench and post the results as soon s possible Nicolas From tziporet at dev.mellanox.co.il Thu Mar 5 02:08:36 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Thu, 05 Mar 2009 12:08:36 +0200 Subject: ***SPAM*** Re: [ofa-general] OFED 1.4.1 is available In-Reply-To: <92daa7bf0903050009t3ce5a969kbe41028164f38ca6@mail.gmail.com> References: <5D49E7A8952DC44FB38C38FA0D758EAD01F50B50@mtlexch01.mtl.com> <92daa7bf0903042151l52d77c20r2440d95c3be49de8@mail.gmail.com> <49AF6D11.70600@gmail.com> <49AF7265.4020506@gmail.com> <49AF7628.5010905@gmail.com> <92daa7bf0903050009t3ce5a969kbe41028164f38ca6@mail.gmail.com> Message-ID: <49AFA4A4.5050505@mellanox.co.il> PN wrote: > I can successfully build the rpm of openmpi by adding the flag. > Thanks. > > Regards, > PN > > > 2009/3/5 vkm > > > Hi, > > JFYI, > > A workaround mentioned here > http://www.open-mpi.org/community/lists/devel/2009/01/5176.php > '--with-contrib-vt-flags=--disable-iotrace' > helps. > > no idea if workaround causes any harm in any way or how good is > it to apply !! ?? > > > Jeff Can this be fixed in Open MPI next RC? Thanks Tziporet From vlad at lists.openfabrics.org Thu Mar 5 03:17:39 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Thu, 5 Mar 2009 03:17:39 -0800 (PST) Subject: [ofa-general] ofa_1_4_kernel 20090305-0200 daily build status Message-ID: <20090305111739.CFD1CE60FAB@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From dorfman.eli at gmail.com Thu Mar 5 04:44:25 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Thu, 05 Mar 2009 14:44:25 +0200 Subject: [ofa-general] ***SPAM*** [PATCH] set IS_SM in local capability mask Message-ID: <49AFC929.20606@gmail.com> set IS_SM in local capability mask Set IS_SM in local portinfo capability mask when setting it on the SMA. This fixes a problem of SM that enters to standby and has mismatch between local capability (IS_SM off) and the SMA. Signed-off-by: Eli Dorfman --- opensm/opensm/osm_port_info_rcv.c | 4 +++- 1 files changed, 3 insertions(+), 1 deletions(-) diff --git a/opensm/opensm/osm_port_info_rcv.c b/opensm/opensm/osm_port_info_rcv.c index 02ad586..d55aef3 100644 --- a/opensm/opensm/osm_port_info_rcv.c +++ b/opensm/opensm/osm_port_info_rcv.c @@ -143,11 +143,13 @@ __osm_pi_rcv_process_endport(IN osm_sm_t * sm, /* We received the PortInfo for our own port. */ - if (!(p_pi->capability_mask & IB_PORT_CAP_IS_SM)) + if (!(p_pi->capability_mask & IB_PORT_CAP_IS_SM)) { + p_physp->port_info.capability_mask |= IB_PORT_CAP_IS_SM; /* Set the IS_SM bit to indicate our port hosts an SM. */ __osm_pi_rcv_set_sm(sm, p_physp); + } } else { p_sm_tbl = &sm->p_subn->sm_guid_tbl; if (p_pi->capability_mask & IB_PORT_CAP_IS_SM) { -- 1.5.5 From hnrose at comcast.net Thu Mar 5 04:24:33 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 5 Mar 2009 07:24:33 -0500 Subject: [ofa-general] [PATCH 1/2] opensm: Handle trap repress on trap 144 generation Message-ID: <20090305122433.GB20341@comcast.net> Make trap generation sending be a transaction (expect a response) and handle incoming trap repress (notice attribute) ib_mad_is_response is now changed to better reflect responses (not just R bit in MAD header) Signed-off-by: Hal Rosenstock --- diff --git a/opensm/include/iba/ib_types.h b/opensm/include/iba/ib_types.h index 5e03265..cf0853e 100644 --- a/opensm/include/iba/ib_types.h +++ b/opensm/include/iba/ib_types.h @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -3783,6 +3784,18 @@ ib_mad_init_response(IN const ib_mad_t * const p_req_mad, * ib_mad_t *********/ +/****d* IBA Base: Constants/IB_BM_ATTR_MOD_RESP +* NAME +* IB_BM_ATTR_MOD_RESP +* +* DESCRIPTION +* Encoded attribute modifier response value used on BM GMPs. +* +* SOURCE +*/ +#define IB_BM_ATTR_MOD_RESP (CL_HTON32(0x000001)) +/**********/ + /****f* IBA Base: Types/ib_mad_is_response * NAME * ib_mad_is_response @@ -3797,8 +3810,10 @@ static inline boolean_t OSM_API ib_mad_is_response(IN const ib_mad_t * const p_mad) { CL_ASSERT(p_mad); - return ((p_mad->method & IB_MAD_METHOD_RESP_MASK) == - IB_MAD_METHOD_RESP_MASK); + return (p_mad->method & IB_MAD_METHOD_RESP_MASK || + p_mad->method == IB_MAD_METHOD_TRAP_REPRESS || + (p_mad->mgmt_class == IB_MCLASS_BM && + p_mad->attr_mod & IB_BM_ATTR_MOD_RESP)); } /* diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c index 2e851b2..1a46e50 100644 --- a/opensm/opensm/osm_req.c +++ b/opensm/opensm/osm_req.c @@ -246,6 +246,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) madw->mad_addr.dest_lid = pi->master_sm_base_lid; madw->mad_addr.addr_type.smi.source_lid = pi->base_lid; + madw->resp_expected = TRUE; madw->fail_msg = CL_DISP_MSGID_NONE; smp = osm_madw_get_smp_ptr(madw); diff --git a/opensm/opensm/osm_sm_mad_ctrl.c b/opensm/opensm/osm_sm_mad_ctrl.c index 267ec85..a9124fa 100644 --- a/opensm/opensm/osm_sm_mad_ctrl.c +++ b/opensm/opensm/osm_sm_mad_ctrl.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -250,10 +251,12 @@ __osm_sm_mad_ctrl_process_get_resp(IN osm_sm_mad_ctrl_t * const p_ctrl, case IB_MAD_ATTR_P_KEY_TABLE: msg_id = OSM_MSG_MAD_PKEY; break; + case IB_MAD_ATTR_NOTICE: + msg_id = OSM_MSG_MAD_NOTICE; + break; case IB_MAD_ATTR_GUID_INFO: case IB_MAD_ATTR_CLASS_PORT_INFO: - case IB_MAD_ATTR_NOTICE: case IB_MAD_ATTR_INFORM_INFO: default: cl_atomic_inc(&p_ctrl->p_stats->qp0_mads_rcvd_unknown); @@ -537,6 +540,57 @@ Exit: * SEE ALSO *********/ +/****f* opensm: SM/__osm_sm_mad_ctrl_process_trap_repress + * NAME + * __osm_sm_mad_ctrl_process_trap_repress + * + * DESCRIPTION + * This function handles method TrapRepress() for received MADs. + * + * SYNOPSIS + */ +static void +__osm_sm_mad_ctrl_process_trap_repress(IN osm_sm_mad_ctrl_t * const p_ctrl, + IN osm_madw_t * p_madw) +{ + ib_smp_t *p_smp; + + OSM_LOG_ENTER(p_ctrl->p_log); + + p_smp = osm_madw_get_smp_ptr(p_madw); + + /* + Note that attr_id (like the rest of the MAD) is in + network byte order. + */ + switch (p_smp->attr_id) { + case IB_MAD_ATTR_NOTICE: + break; + + default: + cl_atomic_inc(&p_ctrl->p_stats->qp0_mads_rcvd_unknown); + OSM_LOG(p_ctrl->p_log, OSM_LOG_ERROR, "ERR 3105: " + "Unsupported attribute = 0x%X\n", + cl_ntoh16(p_smp->attr_id)); + osm_dump_dr_smp(p_ctrl->p_log, p_smp, OSM_LOG_ERROR); + break; + } + + osm_mad_pool_put(p_ctrl->p_mad_pool, p_madw); + + OSM_LOG_EXIT(p_ctrl->p_log); +} + +/* + * PARAMETERS + * + * RETURN VALUES + * + * NOTES + * + * SEE ALSO + *********/ + /****f* opensm: SM/__osm_sm_mad_ctrl_rcv_callback * NAME * __osm_sm_mad_ctrl_rcv_callback @@ -624,10 +678,14 @@ __osm_sm_mad_ctrl_rcv_callback(IN osm_madw_t * p_madw, __osm_sm_mad_ctrl_process_set(p_ctrl, p_madw); break; + case IB_MAD_METHOD_TRAP_REPRESS: + CL_ASSERT(p_req_madw == NULL); + __osm_sm_mad_ctrl_process_trap_repress(p_ctrl, p_madw); + break; + case IB_MAD_METHOD_SEND: case IB_MAD_METHOD_REPORT: case IB_MAD_METHOD_REPORT_RESP: - case IB_MAD_METHOD_TRAP_REPRESS: default: cl_atomic_inc(&p_ctrl->p_stats->qp0_mads_rcvd_unknown); OSM_LOG(p_ctrl->p_log, OSM_LOG_ERROR, "ERR 3112: " diff --git a/opensm/opensm/osm_trap_rcv.c b/opensm/opensm/osm_trap_rcv.c index a5491e1..8a52f1a 100644 --- a/opensm/opensm/osm_trap_rcv.c +++ b/opensm/opensm/osm_trap_rcv.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -649,18 +650,34 @@ Exit: } /********************************************************************** - CURRENTLY WE ARE NOT CREATING TRAPS - SO THIS CALL IN AN ERROR **********************************************************************/ static void trap_rcv_process_response(IN osm_sm_t * sm, IN const osm_madw_t * const p_madw) { + uint8_t payload[sizeof(ib_mad_notice_attr_t)]; + ib_smp_t *p_smp; + ib_mad_notice_attr_t *p_ntci = (ib_mad_notice_attr_t *) payload; + + CL_ASSERT(p_madw); + + if (!osm_log_is_active(sm->p_log, OSM_LOG_VERBOSE)) + return; OSM_LOG_ENTER(sm->p_log); - OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3808: " - "This function is not supported yet\n"); + if (p_madw->p_mad->mgmt_class != IB_MCLASS_SUBN_LID && + p_madw->p_mad->mgmt_class != IB_MCLASS_SUBN_DIR) + goto Exit; + p_smp = osm_madw_get_smp_ptr(p_madw); + + memset(payload, 0, sizeof(payload)); + memcpy(payload, &p_smp->data, IB_SMP_DATA_SIZE); + + osm_dump_notice(sm->p_log, p_ntci, OSM_LOG_VERBOSE); + +Exit: OSM_LOG_EXIT(sm->p_log); } From hnrose at comcast.net Thu Mar 5 04:23:18 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 5 Mar 2009 07:23:18 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] ibsim/sim_mad.c: Add trap/trap repress handling in simulated SMA Message-ID: <20090305122318.GA20341@comcast.net> Add notice attribute support Also, changes to decode_sim_MAD and reply_MAD for trap/trap repress methods Signed-off-by: Hal Rosenstock --- diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c index 32a667e..1a929c5 100644 --- a/ibsim/sim_mad.c +++ b/ibsim/sim_mad.c @@ -57,7 +57,7 @@ typedef int (Smpfn) (Port * port, unsigned op, uint32_t mod, uint8_t * data); typedef int (EncodeTrapfn) (Port * port, char *data); -static Smpfn do_nodeinfo, do_nodedesc, do_switchinfo, do_portinfo, +static Smpfn do_notice, do_nodeinfo, do_nodedesc, do_switchinfo, do_portinfo, do_linearforwtbl, do_multicastforwtbl, do_portcounters, do_extcounters, do_pkeytbl, do_sl2vl, do_vlarb, do_guidinfo, do_cpi; @@ -65,7 +65,8 @@ static EncodeTrapfn encode_trap128; static EncodeTrapfn encode_trap144; static Smpfn *attrs[IB_PERFORMANCE_CLASS + 1][0xff] = { - [IB_SMI_CLASS] {[IB_ATTR_NODE_DESC] do_nodedesc, + [IB_SMI_CLASS] {[NOTICE] do_notice, + [IB_ATTR_NODE_DESC] do_nodedesc, [IB_ATTR_NODE_INFO] do_nodeinfo, [IB_ATTR_SWITCH_INFO] do_switchinfo, [IB_ATTR_PORT_INFO] do_portinfo, @@ -118,15 +119,22 @@ static int decode_sim_MAD(Client * cl, struct sim_request * r, ib_rpc_t * rpc, { void *buf = r->mad; int response; + uint8_t method; - // first word - response = mad_get_field(buf, 0, IB_MAD_RESPONSE_F); + method = mad_get_field(buf, 0, IB_MAD_METHOD_F); + if (method == IB_MAD_METHOD_TRAP_REPRESS) + response = 1; + else if (method == IB_MAD_METHOD_TRAP) + response = 0; + else + // first word + response = mad_get_field(buf, 0, IB_MAD_RESPONSE_F); if (mad_get_field(buf, 0, IB_MAD_CLASSVER_F) > 2 || // sma ver is 1, sa is 2 mad_get_field(buf, 0, IB_MAD_BASEVER_F) != 1) { IBWARN("bad smp headers (1st word)"); return -1; } - rpc->method = mad_get_field(buf, 0, IB_MAD_METHOD_F); + rpc->method = method; rpc->mgtclass = mad_get_field(buf, 0, IB_MAD_MGMTCLASS_F); // second word: @@ -189,11 +197,17 @@ static int forward_MAD(void *buf, ib_rpc_t * rpc, ib_dr_path_t * path) static int reply_MAD(void *buf, ib_rpc_t * rpc, ib_dr_path_t * path, int status, void *data) { - // first word - mad_set_field(buf, 0, IB_MAD_RESPONSE_F, 1); - mad_set_field(buf, 0, IB_MAD_METHOD_F, 0x81); // SUBN_GETRESP - - // second word: + uint8_t method; + + method = mad_get_field(buf, 0, IB_MAD_METHOD_F); + if (method == IB_MAD_METHOD_TRAP) + mad_set_field(buf, 0, IB_MAD_METHOD_F, IB_MAD_METHOD_TRAP_REPRESS); + else if (method != IB_MAD_METHOD_TRAP_REPRESS) { + // first word + mad_set_field(buf, 0, IB_MAD_RESPONSE_F, 1); + mad_set_field(buf, 0, IB_MAD_METHOD_F, 0x81); // SUBN_GETRESP + } + // second word if (rpc->mgtclass == 0x81) { // direct route mad_set_field(buf, 0, IB_DRSMP_STATUS_F, status); mad_set_field(buf, 0, IB_DRSMP_DIRECTION_F, 1); @@ -218,6 +232,16 @@ static int reply_MAD(void *buf, ib_rpc_t * rpc, ib_dr_path_t * path, return 0; } +static int do_notice(Port * port, unsigned op, uint32_t mod, uint8_t * data) +{ + int status = 0; + + if (op != IB_MAD_METHOD_TRAP && op != IB_MAD_METHOD_TRAP_REPRESS) + status = ERR_METHOD_UNSUPPORTED; + + return status; +} + static int do_cpi(Port * port, unsigned op, uint32_t mod, uint8_t * data) { int status = 0; From hnrose at comcast.net Thu Mar 5 04:26:08 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 5 Mar 2009 07:26:08 -0500 Subject: [ofa-general] [PATCH 2/2] libvendor/osm_vendor_mlx_dispatcher.c: Eliminate no longer needed osmv_mad_is_response Message-ID: <20090305122607.GC20341@comcast.net> Use ib_mad_is_response (which now supports trap repress) Signed-off-by: Hal Rosenstock --- diff --git a/opensm/include/vendor/osm_vendor_mlx_svc.h b/opensm/include/vendor/osm_vendor_mlx_svc.h index f23a77d..43a83e3 100644 --- a/opensm/include/vendor/osm_vendor_mlx_svc.h +++ b/opensm/include/vendor/osm_vendor_mlx_svc.h @@ -50,12 +50,6 @@ #endif /* __cplusplus */ BEGIN_C_DECLS -inline static boolean_t osmv_mad_is_response(IN const ib_mad_t * p_mad) -{ - return (ib_mad_is_response(p_mad) || - (p_mad->method == IB_MAD_METHOD_TRAP_REPRESS)); -} - inline static uint8_t osmv_invert_method(IN uint8_t req_method) { switch (req_method) { diff --git a/opensm/libvendor/osm_vendor_ibumad.c b/opensm/libvendor/osm_vendor_ibumad.c index 2d53ff4..f393676 100644 --- a/opensm/libvendor/osm_vendor_ibumad.c +++ b/opensm/libvendor/osm_vendor_ibumad.c @@ -113,7 +113,7 @@ static void clear_madw(osm_vendor_t * p_vend) pthread_mutex_unlock(&p_vend->match_tbl_mutex); OSM_LOG(p_vend->p_log, OSM_LOG_ERROR, "ERR 5401: " "evicting entry %p (tid was 0x%" PRIx64 ")\n", - old_m, old_tid); + old_m, cl_ntoh64(old_tid)); goto Exit; } } diff --git a/opensm/libvendor/osm_vendor_mlx_dispatcher.c b/opensm/libvendor/osm_vendor_mlx_dispatcher.c index d476382..1954547 100644 --- a/opensm/libvendor/osm_vendor_mlx_dispatcher.c +++ b/opensm/libvendor/osm_vendor_mlx_dispatcher.c @@ -193,7 +193,7 @@ __osmv_dispatch_route(IN osm_bind_handle_t h_bind, { ib_api_status_t ret; osmv_bind_obj_t *p_bo = (osmv_bind_obj_t *) h_bind; - boolean_t is_resp = osmv_mad_is_response(p_mad); + boolean_t is_resp = ib_mad_is_response(p_mad); boolean_t is_txn; uint64_t key = cl_ntoh64(p_mad->trans_id); From nicolas.morey-chaisemartin at ext.bull.net Thu Mar 5 04:49:46 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Thu, 05 Mar 2009 13:49:46 +0100 Subject: [ofa-general] [RFC] Fat-Tree upgrades In-Reply-To: <49AF959A.8070907@dev.mellanox.co.il> References: <498FDDD5.1090204@ext.bull.net> <49AE54BF.3080000@dev.mellanox.co.il> <49AE69F4.2080903@ext.bull.net> <49AEA036.2090005@dev.mellanox.co.il> <49AEE40B.9000101@morey-chaisemartin.com> <49AF959A.8070907@dev.mellanox.co.il> Message-ID: <49AFCA6A.7080300@ext.bull.net> Yevgeny Kliteynik wrote: > The speed penalty was my other concern. > Will be happy to see the comparison of the calculation time with and w/o > the sorting once its ready. > I finally managed to achieve "good" performances. I mesured only the routing time using clock() (see patch at the end of the mail) and this command: for i in `seq 1 5`; do opensm -o -a /home_nfs/moreyn/workspace/git/ibsim-t100/config/ftree_root_guids-r8.txt --console local -R ftree; done | grep "Routing time" Network a bit under 4k nodes in ibsim. Topology is a well balanced fat tree, though not completely regular (hence the root guid file). On master (1f5fdf3dfc69733a427520198bcbdd03645bb326): Routing time: 1730000 Routing time: 1710000 Routing time: 1700000 Routing time: 1730000 Routing time: 1720000 On my branch with enhanced fat-tree: Routing time: 1810000 Routing time: 1800000 Routing time: 1790000 Routing time: 1800000 Routing time: 1810000 So we have about 5% loss in routing time which is not so bad. However, as I had to modify my sorting algorithms and tweak a lots of things, I'm still running some non regression tests. Patches shall be posted later in the afternoon if they still work fine. Nicolas ______________________ commit 7db9f3a218d2460fc9826335efdab16cbf73bfb1 Author: Nicolas Morey-Chaisemartin Date: Thu Mar 5 10:00:13 2009 +0100 Added clock tools for timing diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index fffd519..489b788 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -51,6 +51,7 @@ #include #include #include +#include /* * FatTree rank is bounded between 2 and 8: @@ -76,6 +77,9 @@ typedef enum { FTREE_DIRECTION_UP } ftree_direction_t; +clock_t routing_time=0; /*************************************************** ** ** Forward references @@ -3861,7 +3865,7 @@ static int __osm_ftree_do_routing(IN void *context) status = -1; goto Exit; } - + routing_time=clock(); OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, "Starting FatTree routing\n"); @@ -3876,11 +3880,13 @@ static int __osm_ftree_do_routing(IN void *context) OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, "Filling switch forwarding tables for switch-to-switch paths\n"); __osm_ftree_fabric_route_to_switches(p_ftree); - + routing_time=clock()-routing_time; /* for each switch, set its fwd table */ cl_qmap_apply_func(&p_ftree->sw_tbl, __osm_ftree_set_sw_fwd_table, (void *)p_ftree); + printf("Routing time: %lu\n",routing_time); /* write out hca ordering file */ __osm_ftree_fabric_dump_hca_ordering(p_ftree); From dorfman.eli at gmail.com Thu Mar 5 04:59:44 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Thu, 05 Mar 2009 14:59:44 +0200 Subject: [ofa-general] ***SPAM*** [PATCH] fix local port smlid Message-ID: <49AFCCC0.6090507@gmail.com> fix local port smlid when opensm goes from master to standby due to handover its local port smlid will be wrong (equal to its own lid). the new master opensm will change smlid only after local sm (previous master) is in standby and will not query its port again. Signed-off-by: Eli Dorfman --- opensm/opensm/osm_req.c | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-) diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c index 0865ce5..df138d7 100644 --- a/opensm/opensm/osm_req.c +++ b/opensm/opensm/osm_req.c @@ -217,6 +217,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) ib_smp_t *smp; ib_mad_notice_attr_t *ntc; osm_port_t *port; + osm_port_t *smport; ib_port_info_t *pi; port = osm_get_port_by_guid(sm->p_subn, sm->p_subn->sm_port_guid); @@ -229,6 +230,20 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) pi = &port->p_physp->port_info; + smport = osm_get_port_by_guid(sm->p_subn, sm->master_sm_guid); + if (!port) { + OSM_LOG(sm->p_log, OSM_LOG_ERROR, + "ERR 1104: cannot find SM port by guid 0x%" PRIx64 "\n", + cl_ntoh64(sm->master_sm_guid)); + return -1; + } + + /* update local port's smlid - since we may have */ + /* wrong value in local portinfo */ + /* this may happen when we move to standby and master updates */ + /* out smlid but we will not query it */ + pi->master_sm_base_lid = smport->p_physp->port_info.master_sm_base_lid; + /* don't bother with sending trap when SMA supports this */ if (!local && pi->capability_mask&(IB_PORT_CAP_HAS_TRAP|IB_PORT_CAP_HAS_CAP_NTC)) -- 1.5.5 From jsquyres at cisco.com Thu Mar 5 04:57:57 2009 From: jsquyres at cisco.com (Jeff Squyres) Date: Thu, 5 Mar 2009 07:57:57 -0500 Subject: ***SPAM*** Re: [ofa-general] OFED 1.4.1 is available In-Reply-To: <49AFA4A4.5050505@mellanox.co.il> References: <5D49E7A8952DC44FB38C38FA0D758EAD01F50B50@mtlexch01.mtl.com> <92daa7bf0903042151l52d77c20r2440d95c3be49de8@mail.gmail.com> <49AF6D11.70600@gmail.com> <49AF7265.4020506@gmail.com> <49AF7628.5010905@gmail.com> <92daa7bf0903050009t3ce5a969kbe41028164f38ca6@mail.gmail.com> <49AFA4A4.5050505@mellanox.co.il> Message-ID: <803DF940-6CF6-4D9E-B8E6-582A6FF8D97D@cisco.com> On Mar 5, 2009, at 5:08 AM, Tziporet Koren wrote: > Can this be fixed in Open MPI next RC? > Hrm; I thought we had fixed this. I'll check into it. -- Jeff Squyres Cisco Systems From viral.vkm at gmail.com Wed Mar 4 22:34:13 2009 From: viral.vkm at gmail.com (vkm) Date: Thu, 05 Mar 2009 12:04:13 +0530 Subject: ***SPAM*** Re: [ofa-general] OFED 1.4.1 is available In-Reply-To: <49AF6D11.70600@gmail.com> References: <5D49E7A8952DC44FB38C38FA0D758EAD01F50B50@mtlexch01.mtl.com> <92daa7bf0903042151l52d77c20r2440d95c3be49de8@mail.gmail.com> <49AF6D11.70600@gmail.com> Message-ID: <49AF7265.4020506@gmail.com> Hi, A workaround mentioned here http://www.open-mpi.org/community/lists/devel/2009/01/5176.php '--with-contrib-vt-flags=--disable-iotrace' helps. But, certainly would like to know if the workaround causes any harm in any way !! ?? > > > PN wrote: >> HI, >> >> I have a build error of OFED-1.4.1-rc1 under CentOS 5.2: >> >> ..... >> Build openmpi_gcc RPM >> Running rpmbuild --rebuild --define '_topdir /var/tmp/OFED_topdir' >> --define 'dist %{nil}' --target x86_64 --define '_name openmpi_gcc' >> --define 'mpi_selector /usr/bin/mpi-selector' --define >> 'use_mpi_selector 1' --define 'install_shell_scripts 1' --define >> 'shell_scripts_basename mpivars' --define '_usr /usr' --define 'ofed >> 0' --define '_prefix /usr/mpi/gcc/openmpi-1.3.1rc4' --define >> '_defaultdocdir /usr/mpi/gcc/openmpi-1.3.1rc4' --define '_mandir >> %{_prefix}/share/man' --define 'mflags -j 4' --define >> 'configure_options --with-openib=/usr >> --with-openib-libdir=/usr/lib64 CC=gcc CXX=g++ F77=gfortran >> FC=gfortran --enable-mpirun-prefix-by-default' --define >> 'use_default_rpm_opt_flags 1' >> /opt/software/packages/ofed/OFED-1.4.1-rc1/OFED-1.4.1-rc1/SRPMS/openmpi-1.3.1rc4-1.src.rpm >> Failed to build openmpi RPM >> See /tmp/OFED.28377.logs/openmpi.rpmbuild.log >> >> >> In /tmp/OFED.28377.logs/openmpi.rpmbuild.log: >> ..... >> gcc -DHAVE_CONFIG_H -I. -I.. -I../tools/opari/lib >> -I../extlib/otf/otflib -I../extlib/otf/otflib -D_GNU_SOURCE >> -DBINDIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/bin\" >> -DDATADIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/share\" -DRFG -DVT_MEMHOOK >> -DVT_IOWRAP -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions >> -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -MT >> vt_iowrap_helper.o -MD -MP -MF .deps/vt_iowrap_helper.Tpo -c -o >> vt_iowrap_helper.o vt_iowrap_helper.c >> mv -f .deps/vt_memhook.Tpo .deps/vt_memhook.Po >> gcc -DHAVE_CONFIG_H -I. -I.. -I../tools/opari/lib >> -I../extlib/otf/otflib -I../extlib/otf/otflib -D_GNU_SOURCE >> -DBINDIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/bin\" >> -DDATADIR=\"/usr/mpi/gcc/openmpi-1.3.1rc4/share\" -DRFG -DVT_MEMHOOK >> -DVT_IOWRAP -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions >> -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -MT >> rfg_regions.o -MD -MP -MF .deps/rfg_regions.Tpo -c -o rfg_regions.o >> rfg_regions.c >> vt_iowrap.c:1242: error: expected declaration specifiers or '...' >> before numeric constant >> vt_iowrap.c:1243: error: conflicting types for '__fprintf_chk' >> mv -f .deps/vt_iowrap_helper.Tpo .deps/vt_iowrap_helper.Po >> make[5]: *** [vt_iowrap.o] Error 1 >> make[5]: *** Waiting for unfinished jobs.... >> mv -f .deps/vt_comp_gnu.Tpo .deps/vt_comp_gnu.Po >> mv -f .deps/rfg_regions.Tpo .deps/rfg_regions.Po >> make[5]: Leaving directory >> `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt/vt/vtlib' >> make[4]: *** [all-recursive] Error 1 >> make[4]: Leaving directory >> `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt/vt' >> make[3]: *** [all] Error 2 >> make[3]: Leaving directory >> `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt/vt' >> make[2]: *** [all-recursive] Error 1 >> make[2]: Leaving directory >> `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi/contrib/vt' >> make[1]: *** [all-recursive] Error 1 >> make[1]: Leaving directory >> `/var/tmp/OFED_topdir/BUILD/openmpi-1.3.1rc4/ompi' >> make: *** [all-recursive] Error 1 >> error: Bad exit status from /var/tmp/rpm-tmp.40739 (%build) >> >> >> RPM build errors: >> user jsquyres does not exist - using root >> group eng10 does not exist - using root >> user jsquyres does not exist - using root >> group eng10 does not exist - using root >> Bad exit status from /var/tmp/rpm-tmp.40739 (%build) >> >> >> The error seems similar to >> http://www.open-mpi.org/community/lists/devel/2009/01/5230.php >> >> Regards, >> PN >> >> >> 2009/3/5 Tziporet Koren > > >> >> Hi, >> >> OFED-1.4.1-rc1 release is available on >> >> _http://www.openfabrics.org/downloads/OFED/ofed-1.4.1/OFED-1.4.1-rc1.tgz_ >> >> >> To get BUILD_ID run ofed_info >> >> Please report any issues in bugzilla >> _https://bugs.openfabrics.org/_ for OFED 1.4.1 >> >> Vladimir & Tziporet >> >> ======================================================================== >> >> >> Release information: >> >> ------------------------------ >> >> Linux Operating Systems: >> >> - RedHat EL4 up4: 2.6.9-42.ELsmp * >> >> - RedHat EL4 up5: 2.6.9-55.ELsmp >> >> - RedHat EL4 up6: 2.6.9-67.ELsmp >> >> - RedHat EL4 up7: 2.6.9-78.ELsmp >> >> - RedHat EL5: 2.6.18-8.el5 >> >> - RedHat EL5 up1: 2.6.18-53.el5 >> >> - RedHat EL5 up2: 2.6.18-92.el5 >> >> - RedHat EL5 up3: 2.6.18-128.el5 >> >> - OEL 4.5: 2.6.9-55.ELsmp >> >> - OEL 5.2: 2.6.18-92.el5 >> >> - CentOS 5.2: 2.6.18-92.el5 >> >> - Fedora C9: 2.6.25-14.fc9 * >> >> - SLES10: 2.6.16.21-0.8-smp >> >> - SLES10 SP1: 2.6.16.46-0.12-smp >> >> - SLES10 SP1 up1: 2.6.16.53-0.16-smp >> >> - SLES10 SP2: 2.6.16.60-0.21-smp >> >> - SLES11 RC3: 2.6.27.13-1-default >> >> - OpenSuSE 10.3: 2.6.22.5-31 * >> >> - kernel.org : 2.6.26 and >> 2.6.27 >> >> * Minimal QA for these versions >> >> Systems: >> >> * x86_64 >> >> * x86 >> >> * ia64 >> >> * ppc64 >> >> >> Main Changes from OFED-1.4 >> >> ====================== >> >> - Added support for RHEL 5.3 and SLES11 RC3 >> >> - Added RHEL 5.2 backports for NFS/RDMA >> >> - Updated MPI packages: >> >> mvapich-1.1.0-3143 >> >> Open MPI 1.3.1-rc4 >> >> - Updated bonding package: ib-bonding-0.9.0-38 >> >> - Updated opensm version to include critical bug fixes >> >> - Fixed RDS iWARP support >> >> - Low level drivers updated: ehca, mlx4, cxgb3, nes >> >> - Bug fixes >> >> - Attached kernel git tree changes for details >> >> <> >> >> Tasks that should be completed for RC2: >> >> ============================= >> >> 1. High priority bug fixes >> >> 2. RHEL 5.2 32 bits compilation issue (due to NFS/RDMA backports) >> >> 3. Open MPI 1.3.1 release >> >> 4. MVAPICH 1.1.1.minor release with critical bug fixes >> >> 5. OSM release 3.2.6 >> >> 6. Documentation update >> >> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> >> >> >> >> *Email Scanned for Virus & Dangerous Content by :* >> *www.CleanMailGateway.com* >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From hal.rosenstock at gmail.com Thu Mar 5 05:48:29 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 5 Mar 2009 08:48:29 -0500 Subject: ***SPAM*** Re: [ofa-general] ***SPAM*** [PATCH] fix local port smlid In-Reply-To: <49AFCCC0.6090507@gmail.com> References: <49AFCCC0.6090507@gmail.com> Message-ID: On Thu, Mar 5, 2009 at 7:59 AM, Eli Dorfman (Voltaire) wrote: > fix local port smlid > > when opensm goes from master to standby due to handover its local > port smlid will be wrong (equal to its own lid). > the new master opensm will change smlid only after local sm > (previous master) is in standby and will not query its port again. > > Signed-off-by: Eli Dorfman > --- >  opensm/opensm/osm_req.c |   15 +++++++++++++++ >  1 files changed, 15 insertions(+), 0 deletions(-) > > diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c > index 0865ce5..df138d7 100644 > --- a/opensm/opensm/osm_req.c > +++ b/opensm/opensm/osm_req.c > @@ -217,6 +217,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) >        ib_smp_t *smp; >        ib_mad_notice_attr_t *ntc; >        osm_port_t *port; > +       osm_port_t *smport; >        ib_port_info_t *pi; > >        port = osm_get_port_by_guid(sm->p_subn, sm->p_subn->sm_port_guid); > @@ -229,6 +230,20 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) > >        pi = &port->p_physp->port_info; > > +       smport = osm_get_port_by_guid(sm->p_subn, sm->master_sm_guid); > +       if (!port) { smport ? > +               OSM_LOG(sm->p_log, OSM_LOG_ERROR, > +                       "ERR 1104: cannot find SM port by guid 0x%" PRIx64 "\n", nit: 1104 is already used. -- Hal > +                       cl_ntoh64(sm->master_sm_guid)); > +               return -1; > +       } > + > +       /* update local port's smlid - since we may have */ > +       /* wrong value in local portinfo */ > +       /* this may happen when we move to standby and master updates */ > +       /* out smlid but we will not query it */ > +       pi->master_sm_base_lid = smport->p_physp->port_info.master_sm_base_lid; > + >        /* don't bother with sending trap when SMA supports this */ >        if (!local && >            pi->capability_mask&(IB_PORT_CAP_HAS_TRAP|IB_PORT_CAP_HAS_CAP_NTC)) > -- > 1.5.5 > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From nicolas.morey-chaisemartin at ext.bull.net Thu Mar 5 06:00:59 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Thu, 05 Mar 2009 15:00:59 +0100 Subject: [ofa-general] [PATCH 0/3] Enhanced Fat Tree Message-ID: <49AFDB1B.5010004@ext.bull.net> These patches implements the upgrades proposed in a RFC on the mailing list. It involves forcing the min hop route to be used (even if not primary) and equilibrating secondary routes Nicolas Morey-Chaisemartin (3): Enhanced Fat-Tree algorithm: Added balanced minhop within Fat-Tree Added qsort for choosing secondary routes order instead of simple index Enhanced qsort compareason function so it looks at the next switch level. Replaced qsort by enhanced bubble sort opensm/opensm/osm_ucast_ftree.c | 256 +++++++++++++++++++++++++++++---------- 1 files changed, 192 insertions(+), 64 deletions(-) From nicolas.morey-chaisemartin at ext.bull.net Thu Mar 5 06:01:13 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Thu, 05 Mar 2009 15:01:13 +0100 Subject: [ofa-general] [PATCH 1/3] Enhanced Fat-Tree algorithm: Added balanced minhop within Fat-Tree Added qsort for choosing secondary routes order instead of simple index In-Reply-To: References: Message-ID: <49AFDB29.6070402@ext.bull.net> Signed-off-by: Nicolas Morey-Chaisemartin --- opensm/opensm/osm_ucast_ftree.c | 169 ++++++++++++++++++++++++--------------- 1 files changed, 106 insertions(+), 63 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index d92265b..da72ade 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -152,6 +152,7 @@ typedef struct ftree_port_group_t_ { boolean_t is_cn; /* whether this port is a compute node */ boolean_t is_io; /* whether this port is an I/O node */ uint32_t counter_down; /* number of allocated routs downwards */ + uint32_t counter_up; /* number of allocated routs downwards */ } ftree_port_group_t; /*************************************************** @@ -1903,6 +1904,27 @@ static void __osm_ftree_set_sw_fwd_table(IN cl_map_item_t * const p_map_item, p_sw->p_osm_sw); } + + +static int +__osm_ftree_port_group_compare_load_down(const void* p1,const void* p2){ + int temp = (*((ftree_port_group_t**)p1))->counter_down -(*((ftree_port_group_t**)p2))->counter_down ; + if(temp > 0) + return 1; + if(temp < 0) + return -1; + return 0; +} + +static int +__osm_ftree_port_group_compare_load_up(const void* p1,const void* p2){ + int temp = (*((ftree_port_group_t**)p1))->counter_up -(*((ftree_port_group_t**)p2))->counter_up ; + if(temp > 0) + return 1; + if(temp < 0) + return -1; + return 0; +} /*************************************************** ***************************************************/ @@ -1935,10 +1957,10 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, ftree_port_group_t *p_group; ftree_port_t *p_port; ftree_port_t *p_min_port; - uint16_t i; uint16_t j; uint16_t k; boolean_t created_route = FALSE; + boolean_t routed=0; /* we shouldn't enter here if both real_lid and main_path are false */ CL_ASSERT(is_real_lid || is_main_path); @@ -1948,11 +1970,13 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, return FALSE; /* foreach down-going port group (in indexing order) */ - i = p_sw->down_port_groups_idx; + + qsort(p_sw->down_port_groups,p_sw->down_port_groups_num,sizeof(*(p_sw->down_port_groups)), __osm_ftree_port_group_compare_load_up); + + for (k = 0; k < p_sw->down_port_groups_num; k++) { - p_group = p_sw->down_port_groups[i]; - i = (i + 1) % p_sw->down_port_groups_num; + p_group = p_sw->down_port_groups[k]; /* If this port group doesn't point to a switch, mark that the route was created and skip to the next group */ @@ -2002,6 +2026,10 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, cl_ntoh16(p_group->base_lid), __osm_ftree_tuple_to_str(p_sw->tuple), cl_ntoh16(p_group->remote_base_lid)); + /* We skip only if we have come through a longer path */ + if(((target_rank - highest_rank_in_route) + + (p_remote_sw->rank - highest_rank_in_route) + 2*reverse_hops) >= + osm_switch_get_least_hops(p_remote_sw->p_osm_sw, cl_ntoh16(target_lid))) continue; } @@ -2034,10 +2062,6 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, /* second case: skip the port group if the remote (lower) switch has been already configured for this target LID */ - if (is_real_lid && !is_main_path && - p_remote_sw->p_osm_sw->new_lft[cl_ntoh16(target_lid)] != - OSM_NO_PATH) - continue; /* setting fwd tbl port only if this is real LID */ if (is_real_lid) { @@ -2073,19 +2097,26 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, the upper side of the link (on switch with lower rank). Counter is promoted only if we're routing LID on the main path (whether it's a real LID or a dummy one). */ - if (is_main_path) - p_min_port->counter_up++; + /* if (is_main_path) + p_min_port->counter_up++;*/ /* Recursion step: Assign upgoing ports by stepping down, starting on REMOTE switch */ - created_route |= __osm_ftree_fabric_route_upgoing_by_going_down(p_ftree, p_remote_sw, /* remote switch - used as a route-upgoing alg. start point */ - NULL, /* prev. position - NULL to mark that we went down and not up */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - is_real_lid, /* whether the target LID is real or dummy */ - is_main_path, /* whether this is path to HCA that should by tracked by counters */ - highest_rank_in_route, reverse_hops); /* highest visited point in the tree before going down */ - } + routed = + __osm_ftree_fabric_route_upgoing_by_going_down(p_ftree, + p_remote_sw, /* remote switch - used as a route-upgoing alg. start point */ + NULL, /* prev. position - NULL to mark that we went down and not up */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + is_real_lid, /* whether the target LID is real or dummy */ + is_main_path, /* whether this is path to HCA that should by tracked by counters */ + highest_rank_in_route, reverse_hops); /* highest visited point in the tree before going down */ + created_route|=routed; + if(routed) + p_min_port->counter_up++; + p_group->counter_up++; + } + /* done scanning all the down-going port groups */ /* if the route was created, promote the index that @@ -2099,6 +2130,8 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, return created_route; } /* __osm_ftree_fabric_route_upgoing_by_going_down() */ + + /***************************************************/ /* @@ -2112,7 +2145,7 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, * assign-down-going-port-by-ascending-up on REMOTE switch (recursion) */ -static void +static boolean_t __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, IN ftree_sw_t * p_sw, IN ftree_sw_t * p_prev_sw, @@ -2131,12 +2164,14 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, ftree_port_t *p_min_port; uint16_t i; uint16_t j; + boolean_t created_route = FALSE; + boolean_t routed = FALSE; /* we shouldn't enter here if both real_lid and main_path are false */ CL_ASSERT(is_real_lid || is_main_path); /* Assign upgoing ports by stepping down, starting on THIS switch */ - __osm_ftree_fabric_route_upgoing_by_going_down(p_ftree, p_sw, /* local switch - used as a route-upgoing alg. start point */ + created_route = __osm_ftree_fabric_route_upgoing_by_going_down(p_ftree, p_sw, /* local switch - used as a route-upgoing alg. start point */ p_prev_sw, /* switch that we went up from (NULL means that we went down) */ target_lid, /* LID that we're routing to */ target_rank, /* rank of the LID that we're routing to */ @@ -2151,8 +2186,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* We go up by going down as we have some reverse_hop_credit left */ /* We use the index to scatter a bit the reverse up routes */ p_sw->down_port_groups_idx = - (p_sw->down_port_groups_idx + - 1) % p_sw->down_port_groups_num; + (p_sw->down_port_groups_idx + 1) % p_sw->down_port_groups_num; i = p_sw->down_port_groups_idx; for (j = 0; j < p_sw->down_port_groups_num; j++) { @@ -2160,39 +2194,30 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, i = (i + 1) % p_sw->down_port_groups_num; /* Skip this port group unless it points to a switch */ - if (p_group->remote_node_type != - IB_NODE_TYPE_SWITCH) + if (p_group->remote_node_type != IB_NODE_TYPE_SWITCH) continue; p_remote_sw = p_group->remote_hca_or_sw.p_sw; - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ - p_sw, /* this switch - prev. position switch for the function */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - is_real_lid, /* whether this target LID is real or dummy */ - is_main_path, /* whether this is path to HCA that should by tracked by counters */ - reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ - reverse_hops + 1); /* Number of reverse_hops done up to this point */ + created_route |=__osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + p_sw, /* this switch - prev. position switch for the function */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + is_real_lid, /* whether this target LID is real or dummy */ + is_main_path, /* whether this is path to HCA that should by tracked by counters */ + reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ + reverse_hops + 1); /* Number of reverse_hops done up to this point */ } } - return; + return created_route; } - /* Find the least loaded upgoing port group */ - p_min_group = NULL; - for (i = 0; i < p_sw->up_port_groups_num; i++) { - p_group = p_sw->up_port_groups[i]; - if (!p_min_group) { - /* first group that we're checking - use - it as a group with the lowest load */ - p_min_group = p_group; - } else if (p_group->counter_down < p_min_group->counter_down) { - /* this group is less loaded - use it as min */ - p_min_group = p_group; - } - } + /* We should generate a list of port sorted by load so we can find easily the least + * going port and explore the other pots on secondary routes more easily (and quickly) */ + qsort(p_sw->up_port_groups,p_sw->up_port_groups_num,sizeof(*(p_sw->up_port_groups)), __osm_ftree_port_group_compare_load_down); + + p_min_group = p_sw->up_port_groups[0]; /* Find the least loaded upgoing port in the selected group */ p_min_port = NULL; ports_num = (uint16_t) cl_ptr_vector_get_size(&p_min_group->ports); @@ -2297,7 +2322,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* Recursion step: Assign downgoing ports by stepping up, starting on REMOTE switch. */ - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + created_route|=__osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ p_sw, /* this switch - prev. position switch for the function */ target_lid, /* LID that we're routing to */ target_rank, /* rank of the LID that we're routing to */ @@ -2309,7 +2334,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* we're done for the third case */ if (!is_real_lid) - return; + return created_route; /* What's left to do at this point: * @@ -2343,14 +2368,15 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, * - go UP(TRUE,FALSE) to the remote switch */ - for (i = 0; i < p_sw->up_port_groups_num; i++) { + for (i = 1; i < p_sw->up_port_groups_num; i++) { p_group = p_sw->up_port_groups[i]; p_remote_sw = p_group->remote_hca_or_sw.p_sw; - /* skip if target lid has been already set on remote switch fwd tbl */ - if (p_remote_sw->p_osm_sw->new_lft[cl_ntoh16(target_lid)] != - OSM_NO_PATH) - continue; + /* skip if target lid has been already set on remote switch fwd tbl (with a bigger hop count)*/ + if (p_remote_sw->p_osm_sw->new_lft[cl_ntoh16(target_lid)] != OSM_NO_PATH) + if((target_rank -p_remote_sw->rank + 2*reverse_hops) >= + osm_switch_get_least_hops(p_remote_sw->p_osm_sw, cl_ntoh16(target_lid))) + continue; if (p_sw->is_leaf) { OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, @@ -2365,8 +2391,23 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, We can safely assume that switch will initiate very few traffic, so there's no point waisting runtime on trying to balance these routes - always pick port 0. */ + /* GET MIN PORT HERE */ + p_min_port = NULL; + ports_num = (uint16_t) cl_ptr_vector_get_size(&p_group->ports); + for (j = 0; j < ports_num; j++) { + cl_ptr_vector_at(&p_group->ports, j, (void *)&p_port); + if (!p_min_port) { + /* first port that we're checking - use + it as a port with the lowest load */ + p_min_port = p_port; + } else if (p_port->counter_down < p_min_port->counter_down) { + /* this port is less loaded - use it as min */ + p_min_port = p_port; + } + } - cl_ptr_vector_at(&p_group->ports, 0, (void *)&p_port); + p_port = p_min_port; + //cl_ptr_vector_at(&p_group->ports, 0, (void *)&p_port); p_remote_sw->p_osm_sw->new_lft[cl_ntoh16(target_lid)] = p_port->remote_port_num; @@ -2387,7 +2428,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* Recursion step: Assign downgoing ports by stepping up, starting on REMOTE switch. */ - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + routed = __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ p_sw, /* this switch - prev. position switch for the function */ target_lid, /* LID that we're routing to */ target_rank, /* rank of the LID that we're routing to */ @@ -2395,16 +2436,17 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, FALSE, /* whether this is path to HCA that should by tracked by counters */ reverse_hop_credit, /* Remaining reverse_hops allowed */ reverse_hops); /* Number of reverse_hops done up to this point */ + created_route|=routed; } - /* If we don't have any reverse hop credits, we are done */ - if (reverse_hop_credit == 0) - return; + /* If we don't have any reverse hop credits, we are done */ + if(reverse_hop_credit==0) + return created_route; - /* We explore all the down group ports */ - /* We try to reverse jump for each of them */ - /* They already have a route to us from the upgoing_by_going_down started earlier */ - /* This is only so it'll continue exploring up, after this step backwards */ + /* We explore all the down group ports */ + /* We try to reverse jump for each of them */ + /* They already have a route to us from the upgoing_by_going_down started earlier */ + /* This is only so it'll continue exploring up, after this step backwards*/ for (i = 0; i < p_sw->down_port_groups_num; i++) { p_group = p_sw->down_port_groups[i]; p_remote_sw = p_group->remote_hca_or_sw.p_sw; @@ -2415,7 +2457,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* Recursion step: Assign downgoing ports by stepping up, fter doing one step down starting on REMOTE switch. */ - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + created_route |=__osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ p_sw, /* this switch - prev. position switch for the function */ target_lid, /* LID that we're routing to */ target_rank, /* rank of the LID that we're routing to */ @@ -2424,6 +2466,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ reverse_hops + 1); /* Number of reverse_hops done up to this point */ } + return created_route; } /* ftree_fabric_route_downgoing_by_going_up() */ -- 1.6.2-rc2.GIT From nicolas.morey-chaisemartin at ext.bull.net Thu Mar 5 06:01:22 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Thu, 05 Mar 2009 15:01:22 +0100 Subject: [ofa-general] [PATCH 2/3] Enhanced qsort compareason function so it looks at the next switch level. In-Reply-To: References: Message-ID: <49AFDB32.5000506@ext.bull.net> Signed-off-by: Nicolas Morey-Chaisemartin --- opensm/opensm/osm_ucast_ftree.c | 56 ++++++++++++++++++++++++++++++++++++--- 1 files changed, 52 insertions(+), 4 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index da72ade..b00cf4a 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -173,6 +173,7 @@ typedef struct ftree_sw_t_ { uint8_t up_port_groups_num; boolean_t is_leaf; unsigned down_port_groups_idx; + uint32_t min_counter_down; } ftree_sw_t; /*************************************************** @@ -1904,8 +1905,33 @@ static void __osm_ftree_set_sw_fwd_table(IN cl_map_item_t * const p_map_item, p_sw->p_osm_sw); } +static void +__osm_ftree_recalculate_min_counter_down(ftree_sw_t *p_sw){ + uint32_t min= (1<<30); + uint32_t i; + for(i=0;i < p_sw->down_port_groups_num; i++) { + if(p_sw->down_port_groups[i]->counter_down < min){ + min = p_sw->down_port_groups[i]->counter_down; + } + } + p_sw->min_counter_down = min; + return; +} +static uint32_t +__osm_ftree_find_lowest_loaded_group_on_sw(ftree_sw_t *p_sw){ + /*uint32_t min= (1<<30); + uint32_t i; + for(i=0;i < p_sw->down_port_groups_num; i++) { + if(p_sw->down_port_groups[i]->counter_down < min){ + min = p_sw->down_port_groups[i]->counter_down; + } + }*/ + return p_sw->min_counter_down; +} +/* This is for downgoing_by_going_up. + * If load are equals, let's have a look at the remote switches and find the less loaded one */ static int __osm_ftree_port_group_compare_load_down(const void* p1,const void* p2){ int temp = (*((ftree_port_group_t**)p1))->counter_down -(*((ftree_port_group_t**)p2))->counter_down ; @@ -1913,9 +1939,26 @@ __osm_ftree_port_group_compare_load_down(const void* p1,const void* p2){ return 1; if(temp < 0) return -1; - return 0; + + /* Find the less loaded remote sw and choose this one */ + do{ + uint32_t load1=__osm_ftree_find_lowest_loaded_group_on_sw((*((ftree_port_group_t**)p1))->remote_hca_or_sw.p_sw); + uint32_t load2=__osm_ftree_find_lowest_loaded_group_on_sw((*((ftree_port_group_t**)p2))->remote_hca_or_sw.p_sw); + temp = load1-load2; + if(temp > 0) + return 1; + if(temp < 0) + return -1; + }while(0); + + /* If they are both equal, choose the biggest GUID */ + if(((*((ftree_port_group_t**)p1)))->remote_port_guid > ((*((ftree_port_group_t**)p2)))->remote_port_guid) + return 1; + + return -1; } +/* This is for upgoing_by_going_down. There is not much equilibration to do so don't bother looking at the next rank */ static int __osm_ftree_port_group_compare_load_up(const void* p1,const void* p2){ int temp = (*((ftree_port_group_t**)p1))->counter_up -(*((ftree_port_group_t**)p2))->counter_up ; @@ -2216,8 +2259,8 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, * going port and explore the other pots on secondary routes more easily (and quickly) */ qsort(p_sw->up_port_groups,p_sw->up_port_groups_num,sizeof(*(p_sw->up_port_groups)), __osm_ftree_port_group_compare_load_down); - p_min_group = p_sw->up_port_groups[0]; + /* Find the least loaded upgoing port in the selected group */ p_min_port = NULL; ports_num = (uint16_t) cl_ptr_vector_get_size(&p_min_group->ports); @@ -2281,12 +2324,17 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, __osm_ftree_tuple_to_str(p_sw->tuple), __osm_ftree_tuple_to_str(p_remote_sw->tuple)); } + /* The number of downgoing routes is tracked in the p_group->counter_down p_port->counter_down counters of the group and port that belong to the lower side of the link (on switch with higher rank) */ p_min_group->counter_down++; p_min_port->counter_down++; + if(p_min_group->counter_down == (p_min_group->remote_hca_or_sw.p_sw->min_counter_down+1)){ + __osm_ftree_recalculate_min_counter_down(p_min_group->remote_hca_or_sw.p_sw); + } + if (is_real_lid) { /* This LID may already be in the LFT in the reverse_hop feature is used */ /* We update the LFT only if this LID isn't already present. */ @@ -2375,7 +2423,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* skip if target lid has been already set on remote switch fwd tbl (with a bigger hop count)*/ if (p_remote_sw->p_osm_sw->new_lft[cl_ntoh16(target_lid)] != OSM_NO_PATH) if((target_rank -p_remote_sw->rank + 2*reverse_hops) >= - osm_switch_get_least_hops(p_remote_sw->p_osm_sw, cl_ntoh16(target_lid))) + osm_switch_get_least_hops(p_remote_sw->p_osm_sw, cl_ntoh16(target_lid))) continue; if (p_sw->is_leaf) { @@ -2391,7 +2439,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, We can safely assume that switch will initiate very few traffic, so there's no point waisting runtime on trying to balance these routes - always pick port 0. */ - /* GET MIN PORT HERE */ + p_min_port = NULL; ports_num = (uint16_t) cl_ptr_vector_get_size(&p_group->ports); for (j = 0; j < ports_num; j++) { -- 1.6.2-rc2.GIT From nicolas.morey-chaisemartin at ext.bull.net Thu Mar 5 06:01:29 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Thu, 05 Mar 2009 15:01:29 +0100 Subject: [ofa-general] [PATCH 3/3] Replaced qsort by enhanced bubble sort In-Reply-To: References: Message-ID: <49AFDB39.2040500@ext.bull.net> Signed-off-by: Nicolas Morey-Chaisemartin --- opensm/opensm/osm_ucast_ftree.c | 107 ++++++++++++++++++++++++++------------- 1 files changed, 72 insertions(+), 35 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index b00cf4a..501c269 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -81,7 +81,6 @@ typedef enum { ** Forward references ** ***************************************************/ - struct ftree_sw_t_; struct ftree_hca_t_; struct ftree_port_t_; @@ -174,6 +173,7 @@ typedef struct ftree_sw_t_ { boolean_t is_leaf; unsigned down_port_groups_idx; uint32_t min_counter_down; + boolean_t counter_up_changed; } ftree_sw_t; /*************************************************** @@ -1905,7 +1905,7 @@ static void __osm_ftree_set_sw_fwd_table(IN cl_map_item_t * const p_map_item, p_sw->p_osm_sw); } -static void +static inline void __osm_ftree_recalculate_min_counter_down(ftree_sw_t *p_sw){ uint32_t min= (1<<30); uint32_t i; @@ -1918,23 +1918,16 @@ __osm_ftree_recalculate_min_counter_down(ftree_sw_t *p_sw){ return; } -static uint32_t +static inline uint32_t __osm_ftree_find_lowest_loaded_group_on_sw(ftree_sw_t *p_sw){ - /*uint32_t min= (1<<30); - uint32_t i; - for(i=0;i < p_sw->down_port_groups_num; i++) { - if(p_sw->down_port_groups[i]->counter_down < min){ - min = p_sw->down_port_groups[i]->counter_down; - } - }*/ return p_sw->min_counter_down; } /* This is for downgoing_by_going_up. * If load are equals, let's have a look at the remote switches and find the less loaded one */ -static int -__osm_ftree_port_group_compare_load_down(const void* p1,const void* p2){ - int temp = (*((ftree_port_group_t**)p1))->counter_down -(*((ftree_port_group_t**)p2))->counter_down ; +static inline int +__osm_ftree_port_group_compare_load_down(const ftree_port_group_t *p1,const ftree_port_group_t *p2){ + int temp = p1->counter_down -p2->counter_down ; if(temp > 0) return 1; if(temp < 0) @@ -1942,32 +1935,78 @@ __osm_ftree_port_group_compare_load_down(const void* p1,const void* p2){ /* Find the less loaded remote sw and choose this one */ do{ - uint32_t load1=__osm_ftree_find_lowest_loaded_group_on_sw((*((ftree_port_group_t**)p1))->remote_hca_or_sw.p_sw); - uint32_t load2=__osm_ftree_find_lowest_loaded_group_on_sw((*((ftree_port_group_t**)p2))->remote_hca_or_sw.p_sw); + uint32_t load1=__osm_ftree_find_lowest_loaded_group_on_sw(p1->remote_hca_or_sw.p_sw); + uint32_t load2=__osm_ftree_find_lowest_loaded_group_on_sw(p2->remote_hca_or_sw.p_sw); temp = load1-load2; if(temp > 0) return 1; - if(temp < 0) - return -1; + return 0; }while(0); - /* If they are both equal, choose the biggest GUID */ - if(((*((ftree_port_group_t**)p1)))->remote_port_guid > ((*((ftree_port_group_t**)p2)))->remote_port_guid) - return 1; +} - return -1; +static inline void +__osm_ftree_bubble_sort_up(ftree_port_group_t **p_group_array, uint32_t nmemb) +{ + uint32_t i = 0; /* Indice de répétition du tri */ + uint32_t j = 0; /* Variable de boucle */ + ftree_port_group_t *tmp = p_group_array[0]; /* Variable de stockage temporaire et de fin de tri*/ + if(tmp->hca_or_sw.p_sw->counter_up_changed == FALSE){ + return; + } + /* Boucle de répétition du tri et le test qui + arrête le tri dès que le tableau est ordonné */ + for(i = 0 ; tmp ; i++) + { + /* Supposons le tableau ordonné */ + tmp = NULL; + /* Vérification des éléments des places j et j-1 */ + for(j = 1 ; j < (nmemb - i) ; j++) + { + /* Si les 2 éléments sont mal triés */ + if( p_group_array[j]->counter_up < p_group_array[j-1]->counter_up) + { + /* Inversion des 2 éléments */ + tmp = p_group_array[j-1]; + p_group_array[j-1] = p_group_array[j]; + p_group_array[j] = tmp; + + } + } + } + p_group_array[0]->hca_or_sw.p_sw->counter_up_changed = FALSE; } -/* This is for upgoing_by_going_down. There is not much equilibration to do so don't bother looking at the next rank */ -static int -__osm_ftree_port_group_compare_load_up(const void* p1,const void* p2){ - int temp = (*((ftree_port_group_t**)p1))->counter_up -(*((ftree_port_group_t**)p2))->counter_up ; - if(temp > 0) - return 1; - if(temp < 0) - return -1; - return 0; +static inline void +__osm_ftree_bubble_sort_down(ftree_port_group_t **p_group_array, uint32_t nmemb) +{ + uint32_t i = 0; /* Indice de répétition du tri */ + uint32_t j = 0; /* Variable de boucle */ + ftree_port_group_t *tmp = p_group_array[0]; ; /* Variable de stockage temporaire et de fin de tri*/ + + /* Boucle de répétition du tri et le test qui + arrête le tri dès que le tableau est ordonné */ + for(i = 0 ; tmp ; i++) + { + /* Supposons le tableau ordonné */ + tmp = NULL; + /* Vérification des éléments des places j et j-1 */ + for(j = 1 ; j < (nmemb - i) ; j++) + { + /* Si les 2 éléments sont mal triés */ + if( __osm_ftree_port_group_compare_load_down(p_group_array[j],p_group_array[j-1]) < 0 ) + { + /* Inversion des 2 éléments */ + tmp = p_group_array[j-1]; + p_group_array[j-1] = p_group_array[j]; + p_group_array[j] = tmp; + + } + } + } } + + /*************************************************** ***************************************************/ @@ -2013,10 +2052,7 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, return FALSE; /* foreach down-going port group (in indexing order) */ - - qsort(p_sw->down_port_groups,p_sw->down_port_groups_num,sizeof(*(p_sw->down_port_groups)), __osm_ftree_port_group_compare_load_up); - - + __osm_ftree_bubble_sort_up(p_sw->down_port_groups,p_sw->down_port_groups_num); for (k = 0; k < p_sw->down_port_groups_num; k++) { p_group = p_sw->down_port_groups[k]; @@ -2158,6 +2194,7 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, if(routed) p_min_port->counter_up++; p_group->counter_up++; + p_group->hca_or_sw.p_sw->counter_up_changed = TRUE; } /* done scanning all the down-going port groups */ @@ -2257,7 +2294,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* We should generate a list of port sorted by load so we can find easily the least * going port and explore the other pots on secondary routes more easily (and quickly) */ - qsort(p_sw->up_port_groups,p_sw->up_port_groups_num,sizeof(*(p_sw->up_port_groups)), __osm_ftree_port_group_compare_load_down); + __osm_ftree_bubble_sort_down(p_sw->up_port_groups,p_sw->up_port_groups_num); p_min_group = p_sw->up_port_groups[0]; -- 1.6.2-rc2.GIT From dorfman.eli at gmail.com Thu Mar 5 06:29:59 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Thu, 05 Mar 2009 16:29:59 +0200 Subject: ***SPAM*** Re: [ofa-general] [PATCH v2] fix local port smlid In-Reply-To: <49AFCCC0.6090507@gmail.com> References: <49AFCCC0.6090507@gmail.com> Message-ID: <49AFE1E7.3030803@gmail.com> fix local port smlid when opensm goes from master to standby due to handover its local port smlid will be wrong (equal to its own lid). the new master opensm will change local smlid only after local sm (previous master) is in standby and will not query its port again. Update our smlid with master opensm base lid Signed-off-by: Eli Dorfman --- opensm/opensm/osm_req.c | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-) diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c index 0865ce5..df138d7 100644 --- a/opensm/opensm/osm_req.c +++ b/opensm/opensm/osm_req.c @@ -217,6 +217,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) ib_smp_t *smp; ib_mad_notice_attr_t *ntc; osm_port_t *port; + osm_port_t *smport; ib_port_info_t *pi; port = osm_get_port_by_guid(sm->p_subn, sm->p_subn->sm_port_guid); @@ -229,6 +230,20 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) pi = &port->p_physp->port_info; + smport = osm_get_port_by_guid(sm->p_subn, sm->master_sm_guid); + if (!port) { + OSM_LOG(sm->p_log, OSM_LOG_ERROR, + "ERR 1104: cannot find SM port by guid 0x%" PRIx64 "\n", + cl_ntoh64(sm->master_sm_guid)); + return -1; + } + + /* update local port's smlid - since we may have */ + /* wrong value in local portinfo */ + /* this may happen when we move to standby and master updates */ + /* out smlid but we will not query it */ + pi->master_sm_base_lid = smport->p_physp->port_info.base_lid; + /* don't bother with sending trap when SMA supports this */ if (!local && pi->capability_mask&(IB_PORT_CAP_HAS_TRAP|IB_PORT_CAP_HAS_CAP_NTC)) -- 1.5.5 From kliteyn at dev.mellanox.co.il Thu Mar 5 06:27:18 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Thu, 05 Mar 2009 16:27:18 +0200 Subject: [ofa-general] Re: [PATCH 0/3] Enhanced Fat Tree In-Reply-To: <49AFDB1B.5010004@ext.bull.net> References: <49AFDB1B.5010004@ext.bull.net> Message-ID: <49AFE146.2030708@dev.mellanox.co.il> Nicolas Morey Chaisemartin wrote: > These patches implements the upgrades proposed in a RFC on the mailing list. > It involves forcing the min hop route to be used (even if not primary) and equilibrating secondary routes > > Nicolas Morey-Chaisemartin (3): > Enhanced Fat-Tree algorithm: Added balanced minhop within > Fat-Tree Added qsort for choosing secondary routes order > instead of simple index > Enhanced qsort compareason function so it looks at the next switch > level. > Replaced qsort by enhanced bubble sort > > opensm/opensm/osm_ucast_ftree.c | 256 +++++++++++++++++++++++++++++---------- > 1 files changed, 192 insertions(+), 64 deletions(-) > > No patch... :) -- Yevgeny From dorfman.eli at gmail.com Thu Mar 5 06:40:03 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Thu, 05 Mar 2009 16:40:03 +0200 Subject: ***SPAM*** Re: [ofa-general] ***SPAM*** [PATCH] fix local port smlid In-Reply-To: References: <49AFCCC0.6090507@gmail.com> Message-ID: <49AFE443.4030708@gmail.com> Hal Rosenstock wrote: > On Thu, Mar 5, 2009 at 7:59 AM, Eli Dorfman (Voltaire) > wrote: >> fix local port smlid >> >> when opensm goes from master to standby due to handover its local >> port smlid will be wrong (equal to its own lid). >> the new master opensm will change smlid only after local sm >> (previous master) is in standby and will not query its port again. >> >> Signed-off-by: Eli Dorfman >> --- >> opensm/opensm/osm_req.c | 15 +++++++++++++++ >> 1 files changed, 15 insertions(+), 0 deletions(-) >> >> diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c >> index 0865ce5..df138d7 100644 >> --- a/opensm/opensm/osm_req.c >> +++ b/opensm/opensm/osm_req.c >> @@ -217,6 +217,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) >> ib_smp_t *smp; >> ib_mad_notice_attr_t *ntc; >> osm_port_t *port; >> + osm_port_t *smport; >> ib_port_info_t *pi; >> >> port = osm_get_port_by_guid(sm->p_subn, sm->p_subn->sm_port_guid); >> @@ -229,6 +230,20 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) >> >> pi = &port->p_physp->port_info; >> >> + smport = osm_get_port_by_guid(sm->p_subn, sm->master_sm_guid); >> + if (!port) { > > smport ? > meaning master sm port. >> + OSM_LOG(sm->p_log, OSM_LOG_ERROR, >> + "ERR 1104: cannot find SM port by guid 0x%" PRIx64 "\n", > > nit: 1104 is already used. > should i just pick a free number or there is some rules about error numbers? > -- Hal > >> + cl_ntoh64(sm->master_sm_guid)); >> + return -1; >> + } >> + >> + /* update local port's smlid - since we may have */ >> + /* wrong value in local portinfo */ >> + /* this may happen when we move to standby and master updates */ >> + /* out smlid but we will not query it */ >> + pi->master_sm_base_lid = smport->p_physp->port_info.master_sm_base_lid; >> + >> /* don't bother with sending trap when SMA supports this */ >> if (!local && >> pi->capability_mask&(IB_PORT_CAP_HAS_TRAP|IB_PORT_CAP_HAS_CAP_NTC)) >> -- >> 1.5.5 >> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >> From hal.rosenstock at gmail.com Thu Mar 5 06:33:56 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 5 Mar 2009 09:33:56 -0500 Subject: ***SPAM*** Re: ***SPAM*** Re: [ofa-general] [PATCH v2] fix local port smlid In-Reply-To: <49AFE1E7.3030803@gmail.com> References: <49AFCCC0.6090507@gmail.com> <49AFE1E7.3030803@gmail.com> Message-ID: On Thu, Mar 5, 2009 at 9:29 AM, Eli Dorfman (Voltaire) wrote: > fix local port smlid > > when opensm goes from master to standby due to handover its local > port smlid will be wrong (equal to its own lid). > the new master opensm will change local smlid only after local sm > (previous master) is in standby and will not query its port again. > Update our smlid with master opensm base lid > > Signed-off-by: Eli Dorfman > --- >  opensm/opensm/osm_req.c |   15 +++++++++++++++ >  1 files changed, 15 insertions(+), 0 deletions(-) > > diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c > index 0865ce5..df138d7 100644 > --- a/opensm/opensm/osm_req.c > +++ b/opensm/opensm/osm_req.c > @@ -217,6 +217,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) >        ib_smp_t *smp; >        ib_mad_notice_attr_t *ntc; >        osm_port_t *port; > +       osm_port_t *smport; >        ib_port_info_t *pi; > >        port = osm_get_port_by_guid(sm->p_subn, sm->p_subn->sm_port_guid); > @@ -229,6 +230,20 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) > >        pi = &port->p_physp->port_info; > > +       smport = osm_get_port_by_guid(sm->p_subn, sm->master_sm_guid); > +       if (!port) { > +               OSM_LOG(sm->p_log, OSM_LOG_ERROR, > +                       "ERR 1104: cannot find SM port by guid 0x%" PRIx64 "\n", > +                       cl_ntoh64(sm->master_sm_guid)); > +               return -1; > +       } > + > +       /* update local port's smlid - since we may have */ > +       /* wrong value in local portinfo */ > +       /* this may happen when we move to standby and master updates */ > +       /* out smlid but we will not query it */ > +       pi->master_sm_base_lid = smport->p_physp->port_info.base_lid; > + >        /* don't bother with sending trap when SMA supports this */ >        if (!local && >            pi->capability_mask&(IB_PORT_CAP_HAS_TRAP|IB_PORT_CAP_HAS_CAP_NTC)) > -- > 1.5.5 Looks to me like the patch wasn't updated. > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From dorfman.eli at gmail.com Thu Mar 5 06:43:53 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Thu, 05 Mar 2009 16:43:53 +0200 Subject: [ofa-general] ***SPAM*** [PATCH] send trap144 when local priority is higher than master priority Message-ID: <49AFE529.7060506@gmail.com> send trap144 when local priority is higher than master priority send trap144 when local(standby) sm priority is higher than master sm priority. this will trigger heavy sweep by the master and handover of mastership. Signed-off-by: Eli Dorfman --- opensm/opensm/osm_sminfo_rcv.c | 4 ++++ 1 files changed, 4 insertions(+), 0 deletions(-) diff --git a/opensm/opensm/osm_sminfo_rcv.c b/opensm/opensm/osm_sminfo_rcv.c index 98c1994..d0e2ee5 100644 --- a/opensm/opensm/osm_sminfo_rcv.c +++ b/opensm/opensm/osm_sminfo_rcv.c @@ -365,6 +365,10 @@ __osm_sminfo_rcv_process_get_sm(IN osm_sm_t * sm, /* This means the master is alive */ /* Signal that to the SM state mgr */ osm_sm_state_mgr_signal_master_is_alive(sm); + + if (__osm_sminfo_rcv_remote_sm_is_higher + (sm, p_smi) == FALSE) + osm_send_trap144(sm, TRAP_144_MASK_SM_PRIORITY_CHANGE); break; case IB_SMINFO_STATE_STANDBY: /* This should be the response from the sm we are polling. */ -- 1.5.5 From hal.rosenstock at gmail.com Thu Mar 5 06:37:30 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 5 Mar 2009 09:37:30 -0500 Subject: ***SPAM*** Re: [ofa-general] ***SPAM*** [PATCH] fix local port smlid In-Reply-To: <49AFE443.4030708@gmail.com> References: <49AFCCC0.6090507@gmail.com> <49AFE443.4030708@gmail.com> Message-ID: On Thu, Mar 5, 2009 at 9:40 AM, Eli Dorfman (Voltaire) wrote: > - Show quoted text - > Hal Rosenstock wrote: >> On Thu, Mar 5, 2009 at 7:59 AM, Eli Dorfman (Voltaire) >> wrote: >>> fix local port smlid >>> >>> when opensm goes from master to standby due to handover its local >>> port smlid will be wrong (equal to its own lid). >>> the new master opensm will change smlid only after local sm >>> (previous master) is in standby and will not query its port again. >>> >>> Signed-off-by: Eli Dorfman >>> --- >>>  opensm/opensm/osm_req.c |   15 +++++++++++++++ >>>  1 files changed, 15 insertions(+), 0 deletions(-) >>> >>> diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c >>> index 0865ce5..df138d7 100644 >>> --- a/opensm/opensm/osm_req.c >>> +++ b/opensm/opensm/osm_req.c >>> @@ -217,6 +217,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) >>>        ib_smp_t *smp; >>>        ib_mad_notice_attr_t *ntc; >>>        osm_port_t *port; >>> +       osm_port_t *smport; >>>        ib_port_info_t *pi; >>> >>>        port = osm_get_port_by_guid(sm->p_subn, sm->p_subn->sm_port_guid); >>> @@ -229,6 +230,20 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) >>> >>>        pi = &port->p_physp->port_info; >>> >>> +       smport = osm_get_port_by_guid(sm->p_subn, sm->master_sm_guid); >>> +       if (!port) { >> >> smport ? >> > > meaning master sm port. Should the check be if (!smport) rather than if (!port) there ? >>> +               OSM_LOG(sm->p_log, OSM_LOG_ERROR, >>> +                       "ERR 1104: cannot find SM port by guid 0x%" PRIx64 "\n", >> >> nit: 1104 is already used. >> > should i just pick a free number or there is some rules about error numbers? A free number within the module's range (e.g. 1106 or 1103 look like they would work). > - Show quoted text - >> -- Hal >> >>> +                       cl_ntoh64(sm->master_sm_guid)); >>> +               return -1; >>> +       } >>> + >>> +       /* update local port's smlid - since we may have */ >>> +       /* wrong value in local portinfo */ >>> +       /* this may happen when we move to standby and master updates */ >>> +       /* out smlid but we will not query it */ >>> +       pi->master_sm_base_lid = smport->p_physp->port_info.master_sm_base_lid; >>> + >>>        /* don't bother with sending trap when SMA supports this */ >>>        if (!local && >>>            pi->capability_mask&(IB_PORT_CAP_HAS_TRAP|IB_PORT_CAP_HAS_CAP_NTC)) >>> -- >>> 1.5.5 >>> >>> _______________________________________________ >>> general mailing list >>> general at lists.openfabrics.org >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>> >>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >>> > > From dorfman.eli at gmail.com Thu Mar 5 06:49:02 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Thu, 05 Mar 2009 16:49:02 +0200 Subject: ***SPAM*** [ofa-general] [PATCH v3] fix local port smlid In-Reply-To: <49AFE1E7.3030803@gmail.com> References: <49AFCCC0.6090507@gmail.com> <49AFE1E7.3030803@gmail.com> Message-ID: <49AFE65E.3080809@gmail.com> [PATCH] fix local port smlid when opensm goes from master to standby due to handover its local port smlid will be wrong (equal to its own lid). the new master opensm will change smlid only after local sm (previous master) is in standby and will not query its port again. Signed-off-by: Eli Dorfman --- opensm/opensm/osm_req.c | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-) diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c index 0865ce5..df138d7 100644 --- a/opensm/opensm/osm_req.c +++ b/opensm/opensm/osm_req.c @@ -217,6 +217,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) ib_smp_t *smp; ib_mad_notice_attr_t *ntc; osm_port_t *port; + osm_port_t *master_smport; ib_port_info_t *pi; port = osm_get_port_by_guid(sm->p_subn, sm->p_subn->sm_port_guid); @@ -229,6 +230,20 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) pi = &port->p_physp->port_info; + master_smport = osm_get_port_by_guid(sm->p_subn, sm->master_sm_guid); + if (!port) { + OSM_LOG(sm->p_log, OSM_LOG_ERROR, + "ERR 1106: cannot find master SM port by guid 0x%" PRIx64 "\n", + cl_ntoh64(sm->master_sm_guid)); + return -1; + } + + /* update local port's smlid - since we may have */ + /* wrong value in local portinfo */ + /* this may happen when we move to standby and master updates */ + /* out smlid but we will not query it */ + pi->master_sm_base_lid = master_smport->p_physp->port_info.base_lid; + /* don't bother with sending trap when SMA supports this */ if (!local && pi->capability_mask&(IB_PORT_CAP_HAS_TRAP|IB_PORT_CAP_HAS_CAP_NTC)) -- 1.5.5 From ogerlitz at voltaire.com Thu Mar 5 06:45:04 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 5 Mar 2009 16:45:04 +0200 (IST) Subject: [ofa-general] [PATCH] fix offset used for parsing of XmtDataSL & RcvDataSL Message-ID: fix the offset used for parsing the fifth SL of the XmtDataSL & RcvDataSL counters Signed-off-by: Or Gerlitz Index: management/libibmad/src/fields.c =================================================================== --- management.orig/libibmad/src/fields.c +++ management/libibmad/src/fields.c @@ -407,7 +407,7 @@ static const ib_field_t ib_mad_f[] = { {96, 32, "XmtDataSL2", mad_dump_uint}, {128, 32, "XmtDataSL3", mad_dump_uint}, {160, 32, "XmtDataSL4", mad_dump_uint}, - {196, 32, "XmtDataSL5", mad_dump_uint}, + {192, 32, "XmtDataSL5", mad_dump_uint}, {224, 32, "XmtDataSL6", mad_dump_uint}, {256, 32, "XmtDataSL7", mad_dump_uint}, {288, 32, "XmtDataSL8", mad_dump_uint}, @@ -425,7 +425,7 @@ static const ib_field_t ib_mad_f[] = { {96, 32, "RcvDataSL2", mad_dump_uint}, {128, 32, "RcvDataSL3", mad_dump_uint}, {160, 32, "RcvDataSL4", mad_dump_uint}, - {196, 32, "RcvDataSL5", mad_dump_uint}, + {192, 32, "RcvDataSL5", mad_dump_uint}, {224, 32, "RcvDataSL6", mad_dump_uint}, {256, 32, "RcvDataSL7", mad_dump_uint}, {288, 32, "RcvDataSL8", mad_dump_uint}, From hal.rosenstock at gmail.com Thu Mar 5 06:47:42 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 5 Mar 2009 09:47:42 -0500 Subject: ***SPAM*** Re: ***SPAM*** [ofa-general] [PATCH v3] fix local port smlid In-Reply-To: <49AFE65E.3080809@gmail.com> References: <49AFCCC0.6090507@gmail.com> <49AFE1E7.3030803@gmail.com> <49AFE65E.3080809@gmail.com> Message-ID: On Thu, Mar 5, 2009 at 9:49 AM, Eli Dorfman (Voltaire) wrote: > [PATCH] fix local port smlid > > when opensm goes from master to standby due to handover its local > port smlid will be wrong (equal to its own lid). > the new master opensm will change smlid only after local sm > (previous master) is in standby and will not query its port again. > > Signed-off-by: Eli Dorfman > --- >  opensm/opensm/osm_req.c |   15 +++++++++++++++ >  1 files changed, 15 insertions(+), 0 deletions(-) > > diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c > index 0865ce5..df138d7 100644 > --- a/opensm/opensm/osm_req.c > +++ b/opensm/opensm/osm_req.c > @@ -217,6 +217,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) >        ib_smp_t *smp; >        ib_mad_notice_attr_t *ntc; >        osm_port_t *port; > +       osm_port_t *master_smport; >        ib_port_info_t *pi; > >        port = osm_get_port_by_guid(sm->p_subn, sm->p_subn->sm_port_guid); > @@ -229,6 +230,20 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) > >        pi = &port->p_physp->port_info; > > +       master_smport = osm_get_port_by_guid(sm->p_subn, sm->master_sm_guid); > +       if (!port) { if (!master_smport) { > +               OSM_LOG(sm->p_log, OSM_LOG_ERROR, > +                       "ERR 1106: cannot find master SM port by guid 0x%" PRIx64 "\n", > +                       cl_ntoh64(sm->master_sm_guid)); > +               return -1; > +       } > + > +       /* update local port's smlid - since we may have */ > +       /* wrong value in local portinfo */ > +       /* this may happen when we move to standby and master updates */ > +       /* out smlid but we will not query it */ > +       pi->master_sm_base_lid = master_smport->p_physp->port_info.base_lid; > + >        /* don't bother with sending trap when SMA supports this */ >        if (!local && >            pi->capability_mask&(IB_PORT_CAP_HAS_TRAP|IB_PORT_CAP_HAS_CAP_NTC)) > -- > 1.5.5 > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From dorfman.eli at gmail.com Thu Mar 5 06:54:56 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Thu, 05 Mar 2009 16:54:56 +0200 Subject: ***SPAM*** [ofa-general] [PATCH v4] fix local port smlid In-Reply-To: <49AFE65E.3080809@gmail.com> References: <49AFCCC0.6090507@gmail.com> <49AFE1E7.3030803@gmail.com> <49AFE65E.3080809@gmail.com> Message-ID: <49AFE7C0.5060707@gmail.com> fix local port smlid when opensm goes from master to standby due to handover its local port smlid will be wrong (equal to its own lid). the new master opensm will change smlid only after local sm (previous master) is in standby and will not query its port again. Signed-off-by: Eli Dorfman --- opensm/opensm/osm_req.c | 15 +++++++++++++++ 1 files changed, 15 insertions(+), 0 deletions(-) diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c index 0865ce5..df138d7 100644 --- a/opensm/opensm/osm_req.c +++ b/opensm/opensm/osm_req.c @@ -217,6 +217,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) ib_smp_t *smp; ib_mad_notice_attr_t *ntc; osm_port_t *port; + osm_port_t *smport; ib_port_info_t *pi; port = osm_get_port_by_guid(sm->p_subn, sm->p_subn->sm_port_guid); @@ -229,6 +230,20 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) pi = &port->p_physp->port_info; + smport = osm_get_port_by_guid(sm->p_subn, sm->master_sm_guid); + if (!smport) { + OSM_LOG(sm->p_log, OSM_LOG_ERROR, + "ERR 1106: cannot find master SM port by guid 0x%" PRIx64 "\n", + cl_ntoh64(sm->master_sm_guid)); + return -1; + } + + /* update local port's smlid - since we may have */ + /* wrong value in local portinfo */ + /* this may happen when we move to standby and master updates */ + /* out smlid but we will not query it */ + pi->master_sm_base_lid = smport->p_physp->port_info.base_lid; + /* don't bother with sending trap when SMA supports this */ if (!local && pi->capability_mask&(IB_PORT_CAP_HAS_TRAP|IB_PORT_CAP_HAS_CAP_NTC)) -- 1.5.5 From swise at opengridcomputing.com Thu Mar 5 07:36:36 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Thu, 05 Mar 2009 09:36:36 -0600 Subject: [ofa-general] NFS over RDMA In-Reply-To: <49AECBE8.4070707@mellanox.com> References: <49AECBE8.4070707@mellanox.com> Message-ID: <49AFF184.9090406@opengridcomputing.com> If you want to run over iWARP with cxgb3 then I suggest you either try ofed-1.4.1-rc1 or better yet, just use the kernel.org 2.6.28 kernel. The docs in the kernel/Documentation tree are pretty self explanatory. Email me or jon at opengridcomputing.com if you have specific questions/problems. Steve. Vu Pham wrote: > The easiest steps to have nfsrdma up and running: > Download OFED-1.4.1-rc1 from > http://www.openfabrics.org/downloads/OFED/ofed-1.4.1/ > and build/install nfsrdma. It is running on 2.6.27/26/25 and RHEL 5.2 > default kernels > > Or > + Download kernel 2.6.27 from kernel.org > + Build the kernel 2.6.27 with IB driver (and other modules ie. > user_mad, addr_trans, ib_mthca, mlx4_ib... ), nfs client/server, > sunrpc... as modules > + Boot up with new kernel > + Follow the attached README > > NOTE: You can also download and run nfsrdma on latest development > kernel 2.6.29-rc7; however, it does not work on Mellanox ConnectX HCA > due to a bug on Fast memory registration work request. We are > debugging it now > > -vu >> >> Hello, >> >> I want to try NFS over RDMA. According to the documentation enclosed >> within the OFED-1.4 release, it should be as simple as setting >> "nfsrdma=y". >> >> Unfortunately there is no such option. The interactive menu offered by >> './install.pl' does not mention NFS neither. >> >> Is there an updated version of the documentation on how to run NFS over >> RDMA? I have noticed, that it is part of the vanilla Linux kernel but >> I still did >> not manage to get it up and running. >> >> Many thanks for your advice, >> Philip >> >> -- >> Philip Frey IBM Zurich Research Laboratory >> Saumerstrasse 4 | Phone: +41 44 >> 724 8613 >> CH-8803 Rueschlikon/Switzerland | Email: phf at zurich.ibm.com >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general > > ------------------------------------------------------------------------ > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From swise at opengridcomputing.com Thu Mar 5 07:37:44 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Thu, 05 Mar 2009 09:37:44 -0600 Subject: [ofa-general] NFS over RDMA In-Reply-To: <49AECE68.2020703@nasa.gov> References: <49AECBE8.4070707@mellanox.com> <49AECE68.2020703@nasa.gov> Message-ID: <49AFF1C8.2060307@opengridcomputing.com> The stuff shipped in ofed-1.4 isn't very stable. I suggest either ofed-1.4.1-rc1 or just a stock 2.6.28 kernel (see my earlier email). Steve. Jeff Becker wrote: > Hi. If you use install.pl, I think you need to select custom install > (not "install all"). This will allow you to pick the nfsrdma option. > > -jeff > > Vu Pham wrote: > >> The easiest steps to have nfsrdma up and running: >> Download OFED-1.4.1-rc1 from >> http://www.openfabrics.org/downloads/OFED/ofed-1.4.1/ >> and build/install nfsrdma. It is running on 2.6.27/26/25 and RHEL 5.2 >> default kernels >> >> Or >> + Download kernel 2.6.27 from kernel.org >> + Build the kernel 2.6.27 with IB driver (and other modules ie. >> user_mad, addr_trans, ib_mthca, mlx4_ib... ), nfs client/server, >> sunrpc... as modules >> + Boot up with new kernel >> + Follow the attached README >> >> NOTE: You can also download and run nfsrdma on latest development kernel >> 2.6.29-rc7; however, it does not work on Mellanox ConnectX HCA due to a >> bug on Fast memory registration work request. We are debugging it now >> >> -vu >> >> >>> Hello, >>> >>> I want to try NFS over RDMA. According to the documentation enclosed >>> within the OFED-1.4 release, it should be as simple as setting >>> "nfsrdma=y". >>> >>> Unfortunately there is no such option. The interactive menu offered by >>> './install.pl' does not mention NFS neither. >>> >>> Is there an updated version of the documentation on how to run NFS over >>> RDMA? I have noticed, that it is part of the vanilla Linux kernel but >>> I still did >>> not manage to get it up and running. >>> >>> Many thanks for your advice, >>> Philip >>> >>> -- >>> Philip Frey >>> IBM Zurich Research Laboratory >>> Saumerstrasse 4 | Phone: +41 44 >>> 724 8613 >>> CH-8803 Rueschlikon/Switzerland | Email: phf at zurich.ibm.com >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> general mailing list >>> general at lists.openfabrics.org >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>> >>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >>> >>> >> >> > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From eli at dev.mellanox.co.il Thu Mar 5 09:04:13 2009 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Thu, 5 Mar 2009 19:04:13 +0200 Subject: [ofa-general] resolving ipv6 addresses - help needed Message-ID: <20090305170413.GA27571@mtls03> Hi, I am trying to resolve IPv6 addresses (that is, obtaining the MAC address of the interface bearing the IP addresses). I am using rdma_resolve_ip() defined in core/addr.c but I don't get consistent results. I get something like this: fe80::202:c9ff:fe00:1341 dev eth1 FAILED where I would expect to get address to be rachable through eth2. If I run: ping6 -I eth2 fe80::202:c9ff:fe00:1341 then I'd get the address resolved but after some time, when I attempt to resolve again, the operation fails. Is this API tested and reliable? Maybe I'm doing anything wrong... Thanks, Eli From nicolas.morey-chaisemartin at ext.bull.net Thu Mar 5 09:17:10 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Thu, 05 Mar 2009 18:17:10 +0100 Subject: [ofa-general] [PATCH 1/3] Enhanced Fat-Tree algorithm: Added balanced minhop within Fat-Tree Added qsort for choosing secondary routes order instead of simple index In-Reply-To: <49AFDB29.6070402@ext.bull.net> References: <49AFDB29.6070402@ext.bull.net> Message-ID: <49B00916.8000005@ext.bull.net> > > - for (i = 0; i < p_sw->up_port_groups_num; i++) { > + for (i = 1; i < p_sw->up_port_groups_num; i++) { This is wrong, it messes things up in some cases. From sean.hefty at intel.com Thu Mar 5 09:29:48 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 5 Mar 2009 09:29:48 -0800 Subject: [ofa-general] resolving ipv6 addresses - help needed In-Reply-To: <20090305170413.GA27571@mtls03> References: <20090305170413.GA27571@mtls03> Message-ID: >I am trying to resolve IPv6 addresses (that is, obtaining the MAC >address of the interface bearing the IP addresses). I am using >rdma_resolve_ip() defined in core/addr.c but I don't get consistent >results. >I get something like this: > >fe80::202:c9ff:fe00:1341 dev eth1 FAILED > >where I would expect to get address to be rachable through eth2. > >If I run: >ping6 -I eth2 fe80::202:c9ff:fe00:1341 > >then I'd get the address resolved but after some time, when I attempt >to resolve again, the operation fails. I didn't quite follow this last sentence. Are you saying that it only works sometimes? Only the first time? What kernel version are you running? From eli at dev.mellanox.co.il Thu Mar 5 10:36:24 2009 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Thu, 5 Mar 2009 20:36:24 +0200 Subject: [ofa-general] resolving ipv6 addresses - help needed In-Reply-To: References: <20090305170413.GA27571@mtls03> Message-ID: <20090305183624.GA4411@mtls03> On Thu, Mar 05, 2009 at 09:29:48AM -0800, Sean Hefty wrote: > > I didn't quite follow this last sentence. Are you saying that it only works > sometimes? Only the first time? After I run ping6 from the command line, the remote ipv6 address gets added to the cache and resolution starts working. After some time (about 2 sec or so), I try to resolve the same address again but the operation fails. > > What kernel version are you running? 2.6.29-rc6 From yevgenyp at mellanox.co.il Thu Mar 5 11:02:21 2009 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Thu, 05 Mar 2009 21:02:21 +0200 Subject: [ofa-general][PATCH 2/2] mlx4_core: Don't perform SET_PORT command for Eth port from core. In-Reply-To: References: <49AD25DE.9010501@mellanox.co.il> Message-ID: <49B021BD.2060805@mellanox.co.il> Roland Dreier wrote: > I don't really follow this. Is this fixing an existing bug, or is it > something that gets broken if patch 1/2 is applied? > > - R. > The existing code (performing SET_PORT for Ethernet port) is not a bug, bu it is unnecessary. It turns into a bug if patch 1/2 is applied. Thanks, Yevgeny From jgunthorpe at obsidianresearch.com Thu Mar 5 11:11:38 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Thu, 5 Mar 2009 12:11:38 -0700 Subject: [ofa-general] resolving ipv6 addresses - help needed In-Reply-To: <20090305170413.GA27571@mtls03> References: <20090305170413.GA27571@mtls03> Message-ID: <20090305191138.GJ16941@obsidianresearch.com> On Thu, Mar 05, 2009 at 07:04:13PM +0200, Eli Cohen wrote: > Hi, > > I am trying to resolve IPv6 addresses (that is, obtaining the MAC > address of the interface bearing the IP addresses). I am using > rdma_resolve_ip() defined in core/addr.c but I don't get consistent > results. > I get something like this: > > fe80::202:c9ff:fe00:1341 dev eth1 FAILED > > where I would expect to get address to be rachable through eth2. You need to specify a scope for link local addreses. It is actually a bug somewhere along this path that it allowed you to even get as far as you did without specifying a valid scope ID. Ie in normal IP land: $ strace telnet fe80::202:c9ff:fe00:1341 [..] connect(3, {sa_family=AF_INET6, sin6_port=htons(23), inet_pton(AF_INET6, "fe80::202:c9ff:fe00:1341", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 EINVAL (Invalid argument) [..] telnet: Unable to connect to remote host: Invalid argument RMDA should do the same and refuse to start a connection with a scope_id of 0 for link local addresses. It should not choose an address on a random device from the neighbour table. To specify the scope as part of the address use: $ strace telnet fe80::202:c9ff:fe00:1341%eth0 [..] connect(3, {sa_family=AF_INET6, sin6_port=htons(23), inet_pton(AF_INET6, "fe80::202:c9ff:fe00:1341", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=if_nametoindex("eth0")}, 28 And be certain you are using sane multi-family address parsing routings like getnameinfo. Jason From yevgenyp at mellanox.co.il Thu Mar 5 11:39:24 2009 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Thu, 05 Mar 2009 21:39:24 +0200 Subject: [ofa-general][PATCH 1/2] mlx4_core: Link sensing support In-Reply-To: References: <49AD25CE.8010208@mellanox.co.il> Message-ID: <49B02A6C.1000805@mellanox.co.il> Roland Dreier wrote: > > +#include > > this isn't needed, is it? Correct, I should have removed it > > > + queue_delayed_work(sense->sense_wq , &sense->sense_poll, > > + round_jiffies(MLX4_SENSE_RANGE)); > > should be round_jiffies_relative, right? I did not find critical difference between the two: unsigned long round_jiffies(unsigned long j) { return round_jiffies_common(j, raw_smp_processor_id(), false); } unsigned long __round_jiffies_relative(unsigned long j, int cpu) { unsigned long j0 = jiffies; /* Use j0 because jiffies might change while we run */ return round_jiffies_common(j + j0, cpu, false) - j0; } > > > + if (sense->resched) > > Do we need resched? Can't we just use cancel_delayed_work_sync() when > we want to stop this from running? > > > +void mlx4_stop_sense(struct mlx4_dev *dev) > > +{ > > + mlx4_priv(dev)->sense.resched = 0; > > +} > > This doesn't stop anything... we need cancel_delayed_work_sync(). > > > + sense->sense_wq = create_singlethread_workqueue("mlx4_sense"); > You are right about this, and I am using cancel_delayed_work_sync() in mlx4_sense_cleanup(). I was just using the resched flag as an indicator to rearm or not rearm the task, I guess it is redundant and just should use cancel_delayed_work_sync() in mlx4_stop_sense(); > Do we really another work queue, or can we share one queue for the > catastrophic error and port sensing work? There is no special reason for having another queue other then convenience. The catastrophic work queue is defined in catas.c Thanks, Yevgeny From faisal.latif at intel.com Thu Mar 5 12:57:36 2009 From: faisal.latif at intel.com (Faisal Latif) Date: Thu, 5 Mar 2009 14:57:36 -0600 Subject: [ofa-general] [PATCH 1/2] RDMA/nes: Register memory for LSMM Message-ID: <20090305205736.GA10436@flatif-MOBL> The LSMM is using STAG0. Modify the driver to register LSMM and use the non-zero STAG in nes_accept(). Deregister it from nes_disconnect() as it is called during connection termination. Signed-off-by: Faisal Latif --- drivers/infiniband/hw/nes/nes_cm.c | 39 ++++++++++++++++++++++++++++---- drivers/infiniband/hw/nes/nes_verbs.c | 2 + drivers/infiniband/hw/nes/nes_verbs.h | 1 + 3 files changed, 37 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/hw/nes/nes_cm.c b/drivers/infiniband/hw/nes/nes_cm.c index bd918df..6681d57 100644 --- a/drivers/infiniband/hw/nes/nes_cm.c +++ b/drivers/infiniband/hw/nes/nes_cm.c @@ -2486,12 +2486,14 @@ static int nes_disconnect(struct nes_qp *nesqp, int abrupt) int ret = 0; struct nes_vnic *nesvnic; struct nes_device *nesdev; + struct nes_ib_device *nesibdev; nesvnic = to_nesvnic(nesqp->ibqp.device); if (!nesvnic) return -EINVAL; nesdev = nesvnic->nesdev; + nesibdev = nesvnic->nesibdev; nes_debug(NES_DBG_CM, "netdev refcnt = %u.\n", atomic_read(&nesvnic->netdev->refcnt)); @@ -2503,6 +2505,8 @@ static int nes_disconnect(struct nes_qp *nesqp, int abrupt) } else { /* Need to free the Last Streaming Mode Message */ if (nesqp->ietf_frame) { + if (nesqp->lsmm_mr) + nesibdev->ibdev.dereg_mr(nesqp->lsmm_mr); pci_free_consistent(nesdev->pcidev, nesqp->private_data_len+sizeof(struct ietf_mpa_frame), nesqp->ietf_frame, nesqp->ietf_frame_pbase); @@ -2539,6 +2543,12 @@ int nes_accept(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) u32 crc_value; int ret; int passive_state; + struct nes_ib_device *nesibdev; + struct ib_mr *ibmr = NULL; + struct ib_phys_buf ibphysbuf; + struct nes_pd *nespd; + + ibqp = nes_get_qp(cm_id->device, conn_param->qpn); if (!ibqp) @@ -2597,6 +2607,26 @@ int nes_accept(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) if (cm_id->remote_addr.sin_addr.s_addr != cm_id->local_addr.sin_addr.s_addr) { u64temp = (unsigned long)nesqp; + nesibdev = nesvnic->nesibdev; + nespd = nesqp->nespd; + ibphysbuf.addr = nesqp->ietf_frame_pbase; + ibphysbuf.size = conn_param->private_data_len + + sizeof(struct ietf_mpa_frame); + ibmr = nesibdev->ibdev.reg_phys_mr((struct ib_pd *)nespd, + &ibphysbuf, 1, + IB_ACCESS_LOCAL_WRITE, + (u64 *)&nesqp->ietf_frame); + if (!ibmr) { + nes_debug(NES_DBG_CM, "Unable to register memory region" + "for lSMM for cm_node = %p \n", + cm_node); + return -ENOMEM; + } + + ibmr->pd = &nespd->ibpd; + ibmr->device = nespd->ibpd.device; + nesqp->lsmm_mr = ibmr; + u64temp |= NES_SW_CONTEXT_ALIGN>>1; set_wqe_64bit_value(wqe->wqe_words, NES_IWARP_SQ_WQE_COMP_CTX_LOW_IDX, @@ -2607,14 +2637,13 @@ int nes_accept(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) wqe->wqe_words[NES_IWARP_SQ_WQE_TOTAL_PAYLOAD_IDX] = cpu_to_le32(conn_param->private_data_len + sizeof(struct ietf_mpa_frame)); - wqe->wqe_words[NES_IWARP_SQ_WQE_FRAG0_LOW_IDX] = - cpu_to_le32((u32)nesqp->ietf_frame_pbase); - wqe->wqe_words[NES_IWARP_SQ_WQE_FRAG0_HIGH_IDX] = - cpu_to_le32((u32)((u64)nesqp->ietf_frame_pbase >> 32)); + set_wqe_64bit_value(wqe->wqe_words, + NES_IWARP_SQ_WQE_FRAG0_LOW_IDX, + (u64)nesqp->ietf_frame); wqe->wqe_words[NES_IWARP_SQ_WQE_LENGTH0_IDX] = cpu_to_le32(conn_param->private_data_len + sizeof(struct ietf_mpa_frame)); - wqe->wqe_words[NES_IWARP_SQ_WQE_STAG0_IDX] = 0; + wqe->wqe_words[NES_IWARP_SQ_WQE_STAG0_IDX] = ibmr->lkey; nesqp->nesqp_context->ird_ord_sizes |= cpu_to_le32(NES_QPCONTEXT_ORDIRD_LSMM_PRESENT | diff --git a/drivers/infiniband/hw/nes/nes_verbs.c b/drivers/infiniband/hw/nes/nes_verbs.c index b42b17a..6e65c86 100644 --- a/drivers/infiniband/hw/nes/nes_verbs.c +++ b/drivers/infiniband/hw/nes/nes_verbs.c @@ -1363,8 +1363,10 @@ static struct ib_qp *nes_create_qp(struct ib_pd *ibpd, NES_QPCONTEXT_MISC_RQ_SIZE_SHIFT); nesqp->nesqp_context->misc |= cpu_to_le32((u32)nesqp->hwqp.sq_encoded_size << NES_QPCONTEXT_MISC_SQ_SIZE_SHIFT); + if (!udata) { nesqp->nesqp_context->misc |= cpu_to_le32(NES_QPCONTEXT_MISC_PRIV_EN); nesqp->nesqp_context->misc |= cpu_to_le32(NES_QPCONTEXT_MISC_FAST_REGISTER_EN); + } nesqp->nesqp_context->cqs = cpu_to_le32(nesqp->nesscq->hw_cq.cq_number + ((u32)nesqp->nesrcq->hw_cq.cq_number << 16)); u64temp = (u64)nesqp->hwqp.sq_pbase; diff --git a/drivers/infiniband/hw/nes/nes_verbs.h b/drivers/infiniband/hw/nes/nes_verbs.h index da3c368..5e48f67 100644 --- a/drivers/infiniband/hw/nes/nes_verbs.h +++ b/drivers/infiniband/hw/nes/nes_verbs.h @@ -134,6 +134,7 @@ struct nes_qp { struct ietf_mpa_frame *ietf_frame; dma_addr_t ietf_frame_pbase; wait_queue_head_t state_waitq; + struct ib_mr *lsmm_mr; unsigned long socket; struct nes_hw_qp hwqp; struct work_struct work; -- 1.5.3.3 From faisal.latif at intel.com Thu Mar 5 12:59:36 2009 From: faisal.latif at intel.com (Faisal Latif) Date: Thu, 5 Mar 2009 14:59:36 -0600 Subject: [ofa-general] [PATCH 2/2] RDMA/nes: Handle MPA Reject message properly Message-ID: <20090305205936.GA10992@flatif-MOBL> While doing testing, there are failures as MPA Reject call is not handled. To handle MPA Reject call, following changes are done. *Handle inbound/outbound MPA Reject response message. When nes_reject() is called for pending MPA request reply, send the MPA Reject message to its peer (active side)cm_node. The peer cm_node (active side) will indicate Reject message event for the pending Connect Request. *Handle MPA Reject response message for loopback connections and listener. When MPA Request is rejected, check if it is a loopback connection and if it is then it will send Reject message event to its peer loopback node. Also when destroying listener, check if the cm_nodes for that listener are loopback or not. *Add gracefull connection close with the MPA Reject response message. Send gracefull close (FIN, FIN ACK..) to terminate the cm_nodes. *Some code re-org while making the above changes. Removed recv_list and recv_list_lock from the cm_node structure as there can be only one receive close entry on the timer. Also implemented handle_recv_entry() as receive close entry is processed from both nes_rem_ref_cm_node() as well as nes_cm_timer_tick(). Signed-off-by: Faisal Latif --- drivers/infiniband/hw/nes/nes_cm.c | 580 ++++++++++++++++++++++++------------ drivers/infiniband/hw/nes/nes_cm.h | 10 +- 2 files changed, 399 insertions(+), 191 deletions(-) diff --git a/drivers/infiniband/hw/nes/nes_cm.c b/drivers/infiniband/hw/nes/nes_cm.c index 6681d57..5242515 100644 --- a/drivers/infiniband/hw/nes/nes_cm.c +++ b/drivers/infiniband/hw/nes/nes_cm.c @@ -103,6 +103,7 @@ static int nes_disconnect(struct nes_qp *nesqp, int abrupt); static void nes_disconnect_worker(struct work_struct *work); static int send_mpa_request(struct nes_cm_node *, struct sk_buff *); +static int send_mpa_reject(struct nes_cm_node *); static int send_syn(struct nes_cm_node *, u32, struct sk_buff *); static int send_reset(struct nes_cm_node *, struct sk_buff *); static int send_ack(struct nes_cm_node *cm_node, struct sk_buff *skb); @@ -113,8 +114,7 @@ static void process_packet(struct nes_cm_node *, struct sk_buff *, static void active_open_err(struct nes_cm_node *, struct sk_buff *, int); static void passive_open_err(struct nes_cm_node *, struct sk_buff *, int); static void cleanup_retrans_entry(struct nes_cm_node *); -static void handle_rcv_mpa(struct nes_cm_node *, struct sk_buff *, - enum nes_cm_event_type); +static void handle_rcv_mpa(struct nes_cm_node *, struct sk_buff *); static void free_retrans_entry(struct nes_cm_node *cm_node); static int handle_tcp_options(struct nes_cm_node *cm_node, struct tcphdr *tcph, struct sk_buff *skb, int optionsize, int passive); @@ -124,6 +124,8 @@ static void cm_event_connected(struct nes_cm_event *); static void cm_event_connect_error(struct nes_cm_event *); static void cm_event_reset(struct nes_cm_event *); static void cm_event_mpa_req(struct nes_cm_event *); +static void cm_event_mpa_reject(struct nes_cm_event *); +static void handle_recv_entry(struct nes_cm_node *cm_node, u32 rem_node); static void print_core(struct nes_cm_core *core); @@ -196,7 +198,6 @@ static struct nes_cm_event *create_event(struct nes_cm_node *cm_node, */ static int send_mpa_request(struct nes_cm_node *cm_node, struct sk_buff *skb) { - int ret; if (!skb) { nes_debug(NES_DBG_CM, "skb set to NULL\n"); return -1; @@ -206,11 +207,27 @@ static int send_mpa_request(struct nes_cm_node *cm_node, struct sk_buff *skb) form_cm_frame(skb, cm_node, NULL, 0, &cm_node->mpa_frame, cm_node->mpa_frame_size, SET_ACK); - ret = schedule_nes_timer(cm_node, skb, NES_TIMER_TYPE_SEND, 1, 0); - if (ret < 0) - return ret; + return schedule_nes_timer(cm_node, skb, NES_TIMER_TYPE_SEND, 1, 0); +} - return 0; + + +static int send_mpa_reject(struct nes_cm_node *cm_node) +{ + struct sk_buff *skb = NULL; + + skb = dev_alloc_skb(MAX_CM_BUFFER); + if (!skb) { + nes_debug(NES_DBG_CM, "Failed to get a Free pkt\n"); + return -ENOMEM; + } + + /* send an MPA reject frame */ + form_cm_frame(skb, cm_node, NULL, 0, &cm_node->mpa_frame, + cm_node->mpa_frame_size, SET_ACK | SET_FIN); + + cm_node->state = NES_CM_STATE_FIN_WAIT1; + return schedule_nes_timer(cm_node, skb, NES_TIMER_TYPE_SEND, 1, 0); } @@ -218,14 +235,17 @@ static int send_mpa_request(struct nes_cm_node *cm_node, struct sk_buff *skb) * recv_mpa - process a received TCP pkt, we are expecting an * IETF MPA frame */ -static int parse_mpa(struct nes_cm_node *cm_node, u8 *buffer, u32 len) +static int parse_mpa(struct nes_cm_node *cm_node, u8 *buffer, u32 *type, + u32 len) { struct ietf_mpa_frame *mpa_frame; + *type = NES_MPA_REQUEST_ACCEPT; + /* assume req frame is in tcp data payload */ if (len < sizeof(struct ietf_mpa_frame)) { nes_debug(NES_DBG_CM, "The received ietf buffer was too small (%x)\n", len); - return -1; + return -EINVAL; } mpa_frame = (struct ietf_mpa_frame *)buffer; @@ -234,14 +254,25 @@ static int parse_mpa(struct nes_cm_node *cm_node, u8 *buffer, u32 len) if (cm_node->mpa_frame_size + sizeof(struct ietf_mpa_frame) != len) { nes_debug(NES_DBG_CM, "The received ietf buffer was not right" " complete (%x + %x != %x)\n", - cm_node->mpa_frame_size, (u32)sizeof(struct ietf_mpa_frame), len); - return -1; + cm_node->mpa_frame_size, + (u32)sizeof(struct ietf_mpa_frame), len); + return -EINVAL; + } + /* make sure it does not exceed the max size */ + if (len > MAX_CM_BUFFER) { + nes_debug(NES_DBG_CM, "The received ietf buffer was too large" + " (%x + %x != %x)\n", + cm_node->mpa_frame_size, + (u32)sizeof(struct ietf_mpa_frame), len); + return -EINVAL; } /* copy entire MPA frame to our cm_node's frame */ memcpy(cm_node->mpa_frame_buf, buffer + sizeof(struct ietf_mpa_frame), cm_node->mpa_frame_size); + if (mpa_frame->flags & IETF_MPA_FLAGS_REJECT) + *type = NES_MPA_REQUEST_REJECT; return 0; } @@ -380,7 +411,7 @@ int schedule_nes_timer(struct nes_cm_node *cm_node, struct sk_buff *skb, new_send = kzalloc(sizeof(*new_send), GFP_ATOMIC); if (!new_send) - return -1; + return -ENOMEM; /* new_send->timetosend = currenttime */ new_send->retrycount = NES_DEFAULT_RETRYS; @@ -394,9 +425,11 @@ int schedule_nes_timer(struct nes_cm_node *cm_node, struct sk_buff *skb, if (type == NES_TIMER_TYPE_CLOSE) { new_send->timetosend += (HZ/10); - spin_lock_irqsave(&cm_node->recv_list_lock, flags); - list_add_tail(&new_send->list, &cm_node->recv_list); - spin_unlock_irqrestore(&cm_node->recv_list_lock, flags); + if (cm_node->recv_entry) { + WARN_ON(1); + return -EINVAL; + } + cm_node->recv_entry = new_send; } if (type == NES_TIMER_TYPE_SEND) { @@ -435,24 +468,78 @@ int schedule_nes_timer(struct nes_cm_node *cm_node, struct sk_buff *skb, return ret; } +static void nes_retrans_expired(struct nes_cm_node *cm_node) +{ + switch (cm_node->state) { + case NES_CM_STATE_SYN_RCVD: + case NES_CM_STATE_CLOSING: + rem_ref_cm_node(cm_node->cm_core, cm_node); + break; + case NES_CM_STATE_LAST_ACK: + case NES_CM_STATE_FIN_WAIT1: + case NES_CM_STATE_MPAREJ_RCVD: + send_reset(cm_node, NULL); + break; + default: + create_event(cm_node, NES_CM_EVENT_ABORTED); + } +} + +static void handle_recv_entry(struct nes_cm_node *cm_node, u32 rem_node) +{ + struct nes_timer_entry *recv_entry = cm_node->recv_entry; + struct iw_cm_id *cm_id = cm_node->cm_id; + struct nes_qp *nesqp; + unsigned long qplockflags; + + if (!recv_entry) + return; + nesqp = (struct nes_qp *)recv_entry->skb; + if (nesqp) { + spin_lock_irqsave(&nesqp->lock, qplockflags); + if (nesqp->cm_id) { + nes_debug(NES_DBG_CM, "QP%u: cm_id = %p, " + "refcount = %d: HIT A " + "NES_TIMER_TYPE_CLOSE with something " + "to do!!!\n", nesqp->hwqp.qp_id, cm_id, + atomic_read(&nesqp->refcount)); + nesqp->hw_tcp_state = NES_AEQE_TCP_STATE_CLOSED; + nesqp->last_aeq = NES_AEQE_AEID_RESET_SENT; + nesqp->ibqp_state = IB_QPS_ERR; + spin_unlock_irqrestore(&nesqp->lock, qplockflags); + nes_cm_disconn(nesqp); + } else { + spin_unlock_irqrestore(&nesqp->lock, qplockflags); + nes_debug(NES_DBG_CM, "QP%u: cm_id = %p, " + "refcount = %d: HIT A " + "NES_TIMER_TYPE_CLOSE with nothing " + "to do!!!\n", nesqp->hwqp.qp_id, cm_id, + atomic_read(&nesqp->refcount)); + } + } else if (rem_node) { + /* TIME_WAIT state */ + rem_ref_cm_node(cm_node->cm_core, cm_node); + } + if (cm_node->cm_id) + cm_id->rem_ref(cm_id); + kfree(recv_entry); + cm_node->recv_entry = NULL; +} /** * nes_cm_timer_tick */ static void nes_cm_timer_tick(unsigned long pass) { - unsigned long flags, qplockflags; + unsigned long flags; unsigned long nexttimeout = jiffies + NES_LONG_TIME; - struct iw_cm_id *cm_id; struct nes_cm_node *cm_node; struct nes_timer_entry *send_entry, *recv_entry; - struct list_head *list_core, *list_core_temp; - struct list_head *list_node, *list_node_temp; + struct list_head *list_core_temp; + struct list_head *list_node; struct nes_cm_core *cm_core = g_cm_core; - struct nes_qp *nesqp; u32 settimer = 0; int ret = NETDEV_TX_OK; - enum nes_cm_node_state last_state; struct list_head timer_list; INIT_LIST_HEAD(&timer_list); @@ -461,7 +548,7 @@ static void nes_cm_timer_tick(unsigned long pass) list_for_each_safe(list_node, list_core_temp, &cm_core->connected_nodes) { cm_node = container_of(list_node, struct nes_cm_node, list); - if (!list_empty(&cm_node->recv_list) || (cm_node->send_entry)) { + if ((cm_node->recv_entry) || (cm_node->send_entry)) { add_ref_cm_node(cm_node); list_add(&cm_node->timer_entry, &timer_list); } @@ -471,54 +558,18 @@ static void nes_cm_timer_tick(unsigned long pass) list_for_each_safe(list_node, list_core_temp, &timer_list) { cm_node = container_of(list_node, struct nes_cm_node, timer_entry); - spin_lock_irqsave(&cm_node->recv_list_lock, flags); - list_for_each_safe(list_core, list_node_temp, - &cm_node->recv_list) { - recv_entry = container_of(list_core, - struct nes_timer_entry, list); - if (!recv_entry) - break; + recv_entry = cm_node->recv_entry; + + if (recv_entry) { if (time_after(recv_entry->timetosend, jiffies)) { if (nexttimeout > recv_entry->timetosend || - !settimer) { + !settimer) { nexttimeout = recv_entry->timetosend; settimer = 1; } - continue; - } - list_del(&recv_entry->list); - cm_id = cm_node->cm_id; - spin_unlock_irqrestore(&cm_node->recv_list_lock, flags); - nesqp = (struct nes_qp *)recv_entry->skb; - spin_lock_irqsave(&nesqp->lock, qplockflags); - if (nesqp->cm_id) { - nes_debug(NES_DBG_CM, "QP%u: cm_id = %p, " - "refcount = %d: HIT A " - "NES_TIMER_TYPE_CLOSE with something " - "to do!!!\n", nesqp->hwqp.qp_id, cm_id, - atomic_read(&nesqp->refcount)); - nesqp->hw_tcp_state = NES_AEQE_TCP_STATE_CLOSED; - nesqp->last_aeq = NES_AEQE_AEID_RESET_SENT; - nesqp->ibqp_state = IB_QPS_ERR; - spin_unlock_irqrestore(&nesqp->lock, - qplockflags); - nes_cm_disconn(nesqp); - } else { - spin_unlock_irqrestore(&nesqp->lock, - qplockflags); - nes_debug(NES_DBG_CM, "QP%u: cm_id = %p, " - "refcount = %d: HIT A " - "NES_TIMER_TYPE_CLOSE with nothing " - "to do!!!\n", nesqp->hwqp.qp_id, cm_id, - atomic_read(&nesqp->refcount)); - } - if (cm_id) - cm_id->rem_ref(cm_id); - - kfree(recv_entry); - spin_lock_irqsave(&cm_node->recv_list_lock, flags); + } else + handle_recv_entry(cm_node, 1); } - spin_unlock_irqrestore(&cm_node->recv_list_lock, flags); spin_lock_irqsave(&cm_node->retrans_list_lock, flags); do { @@ -533,12 +584,11 @@ static void nes_cm_timer_tick(unsigned long pass) nexttimeout = send_entry->timetosend; settimer = 1; - break; } } else { free_retrans_entry(cm_node); - break; } + break; } if ((cm_node->state == NES_CM_STATE_TSA) || @@ -550,16 +600,12 @@ static void nes_cm_timer_tick(unsigned long pass) if (!send_entry->retranscount || !send_entry->retrycount) { cm_packets_dropped++; - last_state = cm_node->state; - cm_node->state = NES_CM_STATE_CLOSED; free_retrans_entry(cm_node); + spin_unlock_irqrestore( &cm_node->retrans_list_lock, flags); - if (last_state == NES_CM_STATE_SYN_RCVD) - rem_ref_cm_node(cm_core, cm_node); - else - create_event(cm_node, - NES_CM_EVENT_ABORTED); + nes_retrans_expired(cm_node); + cm_node->state = NES_CM_STATE_CLOSED; spin_lock_irqsave(&cm_node->retrans_list_lock, flags); break; @@ -714,7 +760,7 @@ static int send_reset(struct nes_cm_node *cm_node, struct sk_buff *skb) skb = dev_alloc_skb(MAX_CM_BUFFER); if (!skb) { nes_debug(NES_DBG_CM, "Failed to get a Free pkt\n"); - return -1; + return -ENOMEM; } form_cm_frame(skb, cm_node, NULL, 0, NULL, 0, flags); @@ -871,7 +917,8 @@ static int add_hte_node(struct nes_cm_core *cm_core, struct nes_cm_node *cm_node static int mini_cm_dec_refcnt_listen(struct nes_cm_core *cm_core, struct nes_cm_listener *listener, int free_hanging_nodes) { - int ret = 1; + int ret = -EINVAL; + int err = 0; unsigned long flags; struct list_head *list_pos = NULL; struct list_head *list_temp = NULL; @@ -900,10 +947,60 @@ static int mini_cm_dec_refcnt_listen(struct nes_cm_core *cm_core, list_for_each_safe(list_pos, list_temp, &reset_list) { cm_node = container_of(list_pos, struct nes_cm_node, - reset_entry); - cleanup_retrans_entry(cm_node); - send_reset(cm_node, NULL); - rem_ref_cm_node(cm_node->cm_core, cm_node); + reset_entry); + { + struct nes_cm_node *loopback = cm_node->loopbackpartner; + if (NES_CM_STATE_FIN_WAIT1 <= cm_node->state) { + rem_ref_cm_node(cm_node->cm_core, cm_node); + } else { + if (!loopback) { + cleanup_retrans_entry(cm_node); + err = send_reset(cm_node, NULL); + if (err) { + cm_node->state = + NES_CM_STATE_CLOSED; + WARN_ON(1); + } else { + cm_node->state = + NES_CM_STATE_CLOSED; + rem_ref_cm_node( + cm_node->cm_core, + cm_node); + } + } else { + struct nes_cm_event event; + + event.cm_node = loopback; + event.cm_info.rem_addr = + loopback->rem_addr; + event.cm_info.loc_addr = + loopback->loc_addr; + event.cm_info.rem_port = + loopback->rem_port; + event.cm_info.loc_port = + loopback->loc_port; + event.cm_info.cm_id = loopback->cm_id; + cm_event_connect_error(&event); + loopback->state = NES_CM_STATE_CLOSED; + + event.cm_node = cm_node; + event.cm_info.rem_addr = + cm_node->rem_addr; + event.cm_info.loc_addr = + cm_node->loc_addr; + event.cm_info.rem_port = + cm_node->rem_port; + event.cm_info.loc_port = + cm_node->loc_port; + event.cm_info.cm_id = cm_node->cm_id; + cm_event_reset(&event); + + rem_ref_cm_node(cm_node->cm_core, + cm_node); + + } + } + } } spin_lock_irqsave(&cm_core->listen_list_lock, flags); @@ -964,6 +1061,7 @@ static inline int mini_cm_accelerated(struct nes_cm_core *cm_core, if (cm_node->accept_pend) { BUG_ON(!cm_node->listener); atomic_dec(&cm_node->listener->pend_accepts_cnt); + cm_node->accept_pend = 0; BUG_ON(atomic_read(&cm_node->listener->pend_accepts_cnt) < 0); } @@ -990,7 +1088,7 @@ static int nes_addr_resolve_neigh(struct nes_vnic *nesvnic, u32 dst_ip) memset(&fl, 0, sizeof fl); fl.nl_u.ip4_u.daddr = htonl(dst_ip); if (ip_route_output_key(&init_net, &rt, &fl)) { - printk("%s: ip_route_output_key failed for 0x%08X\n", + printk(KERN_ERR "%s: ip_route_output_key failed for 0x%08X\n", __func__, dst_ip); return rc; } @@ -1053,8 +1151,6 @@ static struct nes_cm_node *make_cm_node(struct nes_cm_core *cm_core, cm_node->cm_id); spin_lock_init(&cm_node->retrans_list_lock); - INIT_LIST_HEAD(&cm_node->recv_list); - spin_lock_init(&cm_node->recv_list_lock); cm_node->loopbackpartner = NULL; atomic_set(&cm_node->ref_count, 1); @@ -1122,10 +1218,7 @@ static int add_ref_cm_node(struct nes_cm_node *cm_node) static int rem_ref_cm_node(struct nes_cm_core *cm_core, struct nes_cm_node *cm_node) { - unsigned long flags, qplockflags; - struct nes_timer_entry *recv_entry; - struct iw_cm_id *cm_id; - struct list_head *list_core, *list_node_temp; + unsigned long flags; struct nes_qp *nesqp; if (!cm_node) @@ -1146,38 +1239,9 @@ static int rem_ref_cm_node(struct nes_cm_core *cm_core, atomic_dec(&cm_node->listener->pend_accepts_cnt); BUG_ON(atomic_read(&cm_node->listener->pend_accepts_cnt) < 0); } - BUG_ON(cm_node->send_entry); - spin_lock_irqsave(&cm_node->recv_list_lock, flags); - list_for_each_safe(list_core, list_node_temp, &cm_node->recv_list) { - recv_entry = container_of(list_core, struct nes_timer_entry, - list); - list_del(&recv_entry->list); - cm_id = cm_node->cm_id; - spin_unlock_irqrestore(&cm_node->recv_list_lock, flags); - nesqp = (struct nes_qp *)recv_entry->skb; - spin_lock_irqsave(&nesqp->lock, qplockflags); - if (nesqp->cm_id) { - nes_debug(NES_DBG_CM, "QP%u: cm_id = %p: HIT A " - "NES_TIMER_TYPE_CLOSE with something to do!\n", - nesqp->hwqp.qp_id, cm_id); - nesqp->hw_tcp_state = NES_AEQE_TCP_STATE_CLOSED; - nesqp->last_aeq = NES_AEQE_AEID_RESET_SENT; - nesqp->ibqp_state = IB_QPS_ERR; - spin_unlock_irqrestore(&nesqp->lock, qplockflags); - nes_cm_disconn(nesqp); - } else { - spin_unlock_irqrestore(&nesqp->lock, qplockflags); - nes_debug(NES_DBG_CM, "QP%u: cm_id = %p: HIT A " - "NES_TIMER_TYPE_CLOSE with nothing to do!\n", - nesqp->hwqp.qp_id, cm_id); - } - cm_id->rem_ref(cm_id); - - kfree(recv_entry); - spin_lock_irqsave(&cm_node->recv_list_lock, flags); - } - spin_unlock_irqrestore(&cm_node->recv_list_lock, flags); - + WARN_ON(cm_node->send_entry); + if (cm_node->recv_entry) + handle_recv_entry(cm_node, 0); if (cm_node->listener) { mini_cm_dec_refcnt_listen(cm_core, cm_node->listener, 0); } else { @@ -1262,8 +1326,7 @@ static void drop_packet(struct sk_buff *skb) dev_kfree_skb_any(skb); } -static void handle_fin_pkt(struct nes_cm_node *cm_node, struct sk_buff *skb, - struct tcphdr *tcph) +static void handle_fin_pkt(struct nes_cm_node *cm_node) { nes_debug(NES_DBG_CM, "Received FIN, cm_node = %p, state = %u. " "refcnt=%d\n", cm_node, cm_node->state, @@ -1275,23 +1338,30 @@ static void handle_fin_pkt(struct nes_cm_node *cm_node, struct sk_buff *skb, case NES_CM_STATE_SYN_SENT: case NES_CM_STATE_ESTABLISHED: case NES_CM_STATE_MPAREQ_SENT: + case NES_CM_STATE_MPAREJ_RCVD: cm_node->state = NES_CM_STATE_LAST_ACK; - send_fin(cm_node, skb); + send_fin(cm_node, NULL); break; case NES_CM_STATE_FIN_WAIT1: cm_node->state = NES_CM_STATE_CLOSING; - send_ack(cm_node, skb); + send_ack(cm_node, NULL); + /* Wait for ACK as this is simultanous close.. + * After we receive ACK, do not send anything.. + * Just rm the node.. Done.. */ break; case NES_CM_STATE_FIN_WAIT2: cm_node->state = NES_CM_STATE_TIME_WAIT; - send_ack(cm_node, skb); + send_ack(cm_node, NULL); + schedule_nes_timer(cm_node, NULL, NES_TIMER_TYPE_CLOSE, 1, 0); + break; + case NES_CM_STATE_TIME_WAIT: cm_node->state = NES_CM_STATE_CLOSED; + rem_ref_cm_node(cm_node->cm_core, cm_node); break; case NES_CM_STATE_TSA: default: nes_debug(NES_DBG_CM, "Error Rcvd FIN for node-%p state = %d\n", cm_node, cm_node->state); - drop_packet(skb); break; } } @@ -1337,23 +1407,35 @@ static void handle_rst_pkt(struct nes_cm_node *cm_node, struct sk_buff *skb, cleanup_retrans_entry(cm_node); drop_packet(skb); break; + case NES_CM_STATE_TIME_WAIT: + cleanup_retrans_entry(cm_node); + cm_node->state = NES_CM_STATE_CLOSED; + rem_ref_cm_node(cm_node->cm_core, cm_node); + drop_packet(skb); + break; + case NES_CM_STATE_FIN_WAIT1: + cleanup_retrans_entry(cm_node); + nes_debug(NES_DBG_CM, "Bad state %s[%u]\n", __func__, __LINE__); default: drop_packet(skb); break; } } -static void handle_rcv_mpa(struct nes_cm_node *cm_node, struct sk_buff *skb, - enum nes_cm_event_type type) + +static void handle_rcv_mpa(struct nes_cm_node *cm_node, struct sk_buff *skb) { - int ret; + int ret = 0; int datasize = skb->len; u8 *dataloc = skb->data; - ret = parse_mpa(cm_node, dataloc, datasize); - if (ret < 0) { + + enum nes_cm_event_type type = NES_CM_EVENT_UNKNOWN; + u32 res_type; + ret = parse_mpa(cm_node, dataloc, &res_type, datasize); + if (ret) { nes_debug(NES_DBG_CM, "didn't like MPA Request\n"); - if (type == NES_CM_EVENT_CONNECTED) { + if (cm_node->state == NES_CM_STATE_MPAREQ_SENT) { nes_debug(NES_DBG_CM, "%s[%u] create abort for " "cm_node=%p listener=%p state=%d\n", __func__, __LINE__, cm_node, cm_node->listener, @@ -1362,18 +1444,38 @@ static void handle_rcv_mpa(struct nes_cm_node *cm_node, struct sk_buff *skb, } else { passive_open_err(cm_node, skb, 1); } - } else { - cleanup_retrans_entry(cm_node); - dev_kfree_skb_any(skb); - if (type == NES_CM_EVENT_CONNECTED) + return; + } + + switch (cm_node->state) { + case NES_CM_STATE_ESTABLISHED: + if (res_type == NES_MPA_REQUEST_REJECT) { + /*BIG problem as we are receiving the MPA.. So should + * not be REJECT.. This is Passive Open.. We can + * only receive it Reject for Active Open...*/ + WARN_ON(1); + } + cm_node->state = NES_CM_STATE_MPAREQ_RCVD; + type = NES_CM_EVENT_MPA_REQ; + atomic_set(&cm_node->passive_state, + NES_PASSIVE_STATE_INDICATED); + break; + case NES_CM_STATE_MPAREQ_SENT: + if (res_type == NES_MPA_REQUEST_REJECT) { + type = NES_CM_EVENT_MPA_REJECT; + cm_node->state = NES_CM_STATE_MPAREJ_RCVD; + } else { + type = NES_CM_EVENT_CONNECTED; cm_node->state = NES_CM_STATE_TSA; - else - atomic_set(&cm_node->passive_state, - NES_PASSIVE_STATE_INDICATED); - create_event(cm_node, type); + } + break; + default: + WARN_ON(1); + break; } - return ; + dev_kfree_skb_any(skb); + create_event(cm_node, type); } static void indicate_pkt_err(struct nes_cm_node *cm_node, struct sk_buff *skb) @@ -1461,8 +1563,6 @@ static void handle_syn_pkt(struct nes_cm_node *cm_node, struct sk_buff *skb, break; case NES_CM_STATE_LISTENING: /* Passive OPEN */ - cm_node->accept_pend = 1; - atomic_inc(&cm_node->listener->pend_accepts_cnt); if (atomic_read(&cm_node->listener->pend_accepts_cnt) > cm_node->listener->backlog) { nes_debug(NES_DBG_CM, "drop syn due to backlog " @@ -1480,6 +1580,9 @@ static void handle_syn_pkt(struct nes_cm_node *cm_node, struct sk_buff *skb, } cm_node->tcp_cntxt.rcv_nxt = inc_sequence + 1; BUG_ON(cm_node->send_entry); + cm_node->accept_pend = 1; + atomic_inc(&cm_node->listener->pend_accepts_cnt); + cm_node->state = NES_CM_STATE_SYN_RCVD; send_syn(cm_node, 1, skb); break; @@ -1514,6 +1617,7 @@ static void handle_synack_pkt(struct nes_cm_node *cm_node, struct sk_buff *skb, inc_sequence = ntohl(tcph->seq); switch (cm_node->state) { case NES_CM_STATE_SYN_SENT: + cleanup_retrans_entry(cm_node); /* active open */ if (check_syn(cm_node, tcph, skb)) return; @@ -1563,10 +1667,7 @@ static void handle_ack_pkt(struct nes_cm_node *cm_node, struct sk_buff *skb, u32 rem_seq; int ret; int optionsize; - u32 temp_seq = cm_node->tcp_cntxt.loc_seq_num; - optionsize = (tcph->doff << 2) - sizeof(struct tcphdr); - cm_node->tcp_cntxt.loc_seq_num = ntohl(tcph->ack_seq); if (check_seq(cm_node, tcph, skb)) return; @@ -1576,7 +1677,7 @@ static void handle_ack_pkt(struct nes_cm_node *cm_node, struct sk_buff *skb, rem_seq = ntohl(tcph->seq); rem_seq_ack = ntohl(tcph->ack_seq); datasize = skb->len; - + cleanup_retrans_entry(cm_node); switch (cm_node->state) { case NES_CM_STATE_SYN_RCVD: /* Passive OPEN */ @@ -1584,7 +1685,6 @@ static void handle_ack_pkt(struct nes_cm_node *cm_node, struct sk_buff *skb, if (ret) break; cm_node->tcp_cntxt.rem_ack_num = ntohl(tcph->ack_seq); - cm_node->tcp_cntxt.loc_seq_num = temp_seq; if (cm_node->tcp_cntxt.rem_ack_num != cm_node->tcp_cntxt.loc_seq_num) { nes_debug(NES_DBG_CM, "rem_ack_num != loc_seq_num\n"); @@ -1593,31 +1693,30 @@ static void handle_ack_pkt(struct nes_cm_node *cm_node, struct sk_buff *skb, return; } cm_node->state = NES_CM_STATE_ESTABLISHED; + cleanup_retrans_entry(cm_node); if (datasize) { cm_node->tcp_cntxt.rcv_nxt = inc_sequence + datasize; - cm_node->state = NES_CM_STATE_MPAREQ_RCVD; - handle_rcv_mpa(cm_node, skb, NES_CM_EVENT_MPA_REQ); - } else { /* rcvd ACK only */ + handle_rcv_mpa(cm_node, skb); + } else { /* rcvd ACK only */ dev_kfree_skb_any(skb); cleanup_retrans_entry(cm_node); } break; case NES_CM_STATE_ESTABLISHED: /* Passive OPEN */ - /* We expect mpa frame to be received only */ + cleanup_retrans_entry(cm_node); if (datasize) { cm_node->tcp_cntxt.rcv_nxt = inc_sequence + datasize; - cm_node->state = NES_CM_STATE_MPAREQ_RCVD; - handle_rcv_mpa(cm_node, skb, - NES_CM_EVENT_MPA_REQ); + handle_rcv_mpa(cm_node, skb); } else drop_packet(skb); break; case NES_CM_STATE_MPAREQ_SENT: + cleanup_retrans_entry(cm_node); cm_node->tcp_cntxt.rem_ack_num = ntohl(tcph->ack_seq); if (datasize) { cm_node->tcp_cntxt.rcv_nxt = inc_sequence + datasize; - handle_rcv_mpa(cm_node, skb, NES_CM_EVENT_CONNECTED); + handle_rcv_mpa(cm_node, skb); } else { /* Could be just an ack pkt.. */ cleanup_retrans_entry(cm_node); dev_kfree_skb_any(skb); @@ -1628,13 +1727,24 @@ static void handle_ack_pkt(struct nes_cm_node *cm_node, struct sk_buff *skb, cleanup_retrans_entry(cm_node); send_reset(cm_node, skb); break; + case NES_CM_STATE_LAST_ACK: + cleanup_retrans_entry(cm_node); + cm_node->state = NES_CM_STATE_CLOSED; + cm_node->cm_id->rem_ref(cm_node->cm_id); + case NES_CM_STATE_CLOSING: + cleanup_retrans_entry(cm_node); + rem_ref_cm_node(cm_node->cm_core, cm_node); + drop_packet(skb); + break; case NES_CM_STATE_FIN_WAIT1: + cleanup_retrans_entry(cm_node); + drop_packet(skb); + cm_node->state = NES_CM_STATE_FIN_WAIT2; + break; case NES_CM_STATE_SYN_SENT: case NES_CM_STATE_FIN_WAIT2: case NES_CM_STATE_TSA: case NES_CM_STATE_MPAREQ_RCVD: - case NES_CM_STATE_LAST_ACK: - case NES_CM_STATE_CLOSING: case NES_CM_STATE_UNKNOWN: default: drop_packet(skb); @@ -1744,6 +1854,7 @@ static void process_packet(struct nes_cm_node *cm_node, struct sk_buff *skb, { enum nes_tcpip_pkt_type pkt_type = NES_PKT_TYPE_UNKNOWN; struct tcphdr *tcph = tcp_hdr(skb); + u32 fin_set = 0; skb_pull(skb, ip_hdr(skb)->ihl << 2); nes_debug(NES_DBG_CM, "process_packet: cm_node=%p state =%d syn=%d " @@ -1756,10 +1867,10 @@ static void process_packet(struct nes_cm_node *cm_node, struct sk_buff *skb, pkt_type = NES_PKT_TYPE_SYN; if (tcph->ack) pkt_type = NES_PKT_TYPE_SYNACK; - } else if (tcph->fin) - pkt_type = NES_PKT_TYPE_FIN; - else if (tcph->ack) + } else if (tcph->ack) pkt_type = NES_PKT_TYPE_ACK; + if (tcph->fin) + fin_set = 1; switch (pkt_type) { case NES_PKT_TYPE_SYN: @@ -1770,15 +1881,16 @@ static void process_packet(struct nes_cm_node *cm_node, struct sk_buff *skb, break; case NES_PKT_TYPE_ACK: handle_ack_pkt(cm_node, skb, tcph); + if (fin_set) + handle_fin_pkt(cm_node); break; case NES_PKT_TYPE_RST: handle_rst_pkt(cm_node, skb, tcph); break; - case NES_PKT_TYPE_FIN: - handle_fin_pkt(cm_node, skb, tcph); - break; default: drop_packet(skb); + if (fin_set) + handle_fin_pkt(cm_node); break; } } @@ -1921,7 +2033,7 @@ static struct nes_cm_node *mini_cm_connect(struct nes_cm_core *cm_core, loopbackremotenode->tcp_cntxt.rcv_wscale; loopbackremotenode->tcp_cntxt.snd_wscale = cm_node->tcp_cntxt.rcv_wscale; - + loopbackremotenode->state = NES_CM_STATE_MPAREQ_RCVD; create_event(loopbackremotenode, NES_CM_EVENT_MPA_REQ); } return cm_node; @@ -1976,7 +2088,11 @@ static int mini_cm_reject(struct nes_cm_core *cm_core, struct ietf_mpa_frame *mpa_frame, struct nes_cm_node *cm_node) { int ret = 0; + int err = 0; int passive_state; + struct nes_cm_event event; + struct iw_cm_id *cm_id = cm_node->cm_id; + struct nes_cm_node *loopback = cm_node->loopbackpartner; nes_debug(NES_DBG_CM, "%s cm_node=%p type=%d state=%d\n", __func__, cm_node, cm_node->tcp_cntxt.client, cm_node->state); @@ -1985,12 +2101,38 @@ static int mini_cm_reject(struct nes_cm_core *cm_core, return ret; cleanup_retrans_entry(cm_node); - passive_state = atomic_add_return(1, &cm_node->passive_state); - cm_node->state = NES_CM_STATE_CLOSED; - if (passive_state == NES_SEND_RESET_EVENT) + if (!loopback) { + passive_state = atomic_add_return(1, &cm_node->passive_state); + if (passive_state == NES_SEND_RESET_EVENT) { + cm_node->state = NES_CM_STATE_CLOSED; + rem_ref_cm_node(cm_core, cm_node); + } else { + ret = send_mpa_reject(cm_node); + if (ret) { + cm_node->state = NES_CM_STATE_CLOSED; + err = send_reset(cm_node, NULL); + if (err) + WARN_ON(1); + } else + cm_id->add_ref(cm_id); + } + } else { + cm_node->cm_id = NULL; + event.cm_node = loopback; + event.cm_info.rem_addr = loopback->rem_addr; + event.cm_info.loc_addr = loopback->loc_addr; + event.cm_info.rem_port = loopback->rem_port; + event.cm_info.loc_port = loopback->loc_port; + event.cm_info.cm_id = loopback->cm_id; + cm_event_mpa_reject(&event); rem_ref_cm_node(cm_core, cm_node); - else - ret = send_reset(cm_node, NULL); + loopback->state = NES_CM_STATE_CLOSING; + + cm_id = loopback->cm_id; + rem_ref_cm_node(cm_core, loopback); + cm_id->rem_ref(cm_id); + } + return ret; } @@ -2027,6 +2169,7 @@ static int mini_cm_close(struct nes_cm_core *cm_core, struct nes_cm_node *cm_nod case NES_CM_STATE_CLOSING: ret = -1; break; + case NES_CM_STATE_MPAREJ_RCVD: case NES_CM_STATE_LISTENING: case NES_CM_STATE_UNKNOWN: case NES_CM_STATE_INITED: @@ -2223,15 +2366,15 @@ static int mini_cm_set(struct nes_cm_core *cm_core, u32 type, u32 value) int ret = 0; switch (type) { - case NES_CM_SET_PKT_SIZE: - cm_core->mtu = value; - break; - case NES_CM_SET_FREE_PKT_Q_SIZE: - cm_core->free_tx_pkt_max = value; - break; - default: - /* unknown set option */ - ret = -EINVAL; + case NES_CM_SET_PKT_SIZE: + cm_core->mtu = value; + break; + case NES_CM_SET_FREE_PKT_Q_SIZE: + cm_core->free_tx_pkt_max = value; + break; + default: + /* unknown set option */ + ret = -EINVAL; } return ret; @@ -2650,9 +2793,7 @@ int nes_accept(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) NES_QPCONTEXT_ORDIRD_WRPDU); } else { nesqp->nesqp_context->ird_ord_sizes |= - cpu_to_le32((NES_QPCONTEXT_ORDIRD_LSMM_PRESENT | - NES_QPCONTEXT_ORDIRD_WRPDU | - NES_QPCONTEXT_ORDIRD_ALSMM)); + cpu_to_le32(NES_QPCONTEXT_ORDIRD_WRPDU); } nesqp->skip_lsmm = 1; @@ -2774,23 +2915,35 @@ int nes_accept(struct iw_cm_id *cm_id, struct iw_cm_conn_param *conn_param) int nes_reject(struct iw_cm_id *cm_id, const void *pdata, u8 pdata_len) { struct nes_cm_node *cm_node; + struct nes_cm_node *loopback; + struct nes_cm_core *cm_core; atomic_inc(&cm_rejects); cm_node = (struct nes_cm_node *) cm_id->provider_data; + loopback = cm_node->loopbackpartner; cm_core = cm_node->cm_core; + cm_node->cm_id = cm_id; cm_node->mpa_frame_size = sizeof(struct ietf_mpa_frame) + pdata_len; + if (cm_node->mpa_frame_size > MAX_CM_BUFFER) + return -EINVAL; + strcpy(&cm_node->mpa_frame.key[0], IEFT_MPA_KEY_REP); - memcpy(&cm_node->mpa_frame.priv_data, pdata, pdata_len); + if (loopback) { + memcpy(&loopback->mpa_frame.priv_data, pdata, pdata_len); + loopback->mpa_frame.priv_data_len = pdata_len; + loopback->mpa_frame_size = sizeof(struct ietf_mpa_frame) + + pdata_len; + } else { + memcpy(&cm_node->mpa_frame.priv_data, pdata, pdata_len); + cm_node->mpa_frame.priv_data_len = cpu_to_be16(pdata_len); + } - cm_node->mpa_frame.priv_data_len = cpu_to_be16(pdata_len); cm_node->mpa_frame.rev = mpa_version; cm_node->mpa_frame.flags = IETF_MPA_FLAGS_CRC | IETF_MPA_FLAGS_REJECT; - cm_core->api->reject(cm_core, &cm_node->mpa_frame, cm_node); - - return 0; + return cm_core->api->reject(cm_core, &cm_node->mpa_frame, cm_node); } @@ -3299,13 +3452,56 @@ static void cm_event_mpa_req(struct nes_cm_event *event) cm_event.remote_addr.sin_family = AF_INET; cm_event.remote_addr.sin_port = htons(event->cm_info.rem_port); cm_event.remote_addr.sin_addr.s_addr = htonl(event->cm_info.rem_addr); + cm_event.private_data = cm_node->mpa_frame_buf; + cm_event.private_data_len = (u8) cm_node->mpa_frame_size; + + ret = cm_id->event_handler(cm_id, &cm_event); + if (ret) + printk(KERN_ERR "%s[%u] OFA CM event_handler returned, ret=%d\n", + __func__, __LINE__, ret); + return; +} + + +static void cm_event_mpa_reject(struct nes_cm_event *event) +{ + struct iw_cm_id *cm_id; + struct iw_cm_event cm_event; + struct nes_cm_node *cm_node; + int ret; + + cm_node = event->cm_node; + if (!cm_node) + return; + cm_id = cm_node->cm_id; + + atomic_inc(&cm_connect_reqs); + nes_debug(NES_DBG_CM, "cm_node = %p - cm_id = %p, jiffies = %lu\n", + cm_node, cm_id, jiffies); + + cm_event.event = IW_CM_EVENT_CONNECT_REPLY; + cm_event.status = -ECONNREFUSED; + cm_event.provider_data = cm_id->provider_data; + + cm_event.local_addr.sin_family = AF_INET; + cm_event.local_addr.sin_port = htons(event->cm_info.loc_port); + cm_event.local_addr.sin_addr.s_addr = htonl(event->cm_info.loc_addr); + + cm_event.remote_addr.sin_family = AF_INET; + cm_event.remote_addr.sin_port = htons(event->cm_info.rem_port); + cm_event.remote_addr.sin_addr.s_addr = htonl(event->cm_info.rem_addr); - cm_event.private_data = cm_node->mpa_frame_buf; - cm_event.private_data_len = (u8) cm_node->mpa_frame_size; + cm_event.private_data = cm_node->mpa_frame_buf; + cm_event.private_data_len = (u8) cm_node->mpa_frame_size; + + nes_debug(NES_DBG_CM, "call CM_EVENT_MPA_REJECTED, local_addr=%08x, " + "remove_addr=%08x\n", + cm_event.local_addr.sin_addr.s_addr, + cm_event.remote_addr.sin_addr.s_addr); ret = cm_id->event_handler(cm_id, &cm_event); if (ret) - printk("%s[%u] OFA CM event_handler returned, ret=%d\n", + printk(KERN_ERR "%s[%u] OFA CM event_handler returned, ret=%d\n", __func__, __LINE__, ret); return; @@ -3370,6 +3566,14 @@ static void nes_cm_event_handler(struct work_struct *work) cm_event_connected(event); nes_debug(NES_DBG_CM, "CM Event: CONNECTED\n"); break; + case NES_CM_EVENT_MPA_REJECT: + if ((!event->cm_node->cm_id) || + (event->cm_node->state == NES_CM_STATE_TSA)) + break; + cm_event_mpa_reject(event); + nes_debug(NES_DBG_CM, "CM Event: REJECT\n"); + break; + case NES_CM_EVENT_ABORTED: if ((!event->cm_node->cm_id) || (event->cm_node->state == NES_CM_STATE_TSA)) diff --git a/drivers/infiniband/hw/nes/nes_cm.h b/drivers/infiniband/hw/nes/nes_cm.h index 4ab2beb..d5f7782 100644 --- a/drivers/infiniband/hw/nes/nes_cm.h +++ b/drivers/infiniband/hw/nes/nes_cm.h @@ -39,6 +39,9 @@ #define NES_MANAGE_APBVT_DEL 0 #define NES_MANAGE_APBVT_ADD 1 +#define NES_MPA_REQUEST_ACCEPT 1 +#define NES_MPA_REQUEST_REJECT 2 + /* IETF MPA -- defines, enums, structs */ #define IEFT_MPA_KEY_REQ "MPA ID Req Frame" #define IEFT_MPA_KEY_REP "MPA ID Rep Frame" @@ -186,6 +189,7 @@ enum nes_cm_node_state { NES_CM_STATE_ACCEPTING, NES_CM_STATE_MPAREQ_SENT, NES_CM_STATE_MPAREQ_RCVD, + NES_CM_STATE_MPAREJ_RCVD, NES_CM_STATE_TSA, NES_CM_STATE_FIN_WAIT1, NES_CM_STATE_FIN_WAIT2, @@ -278,13 +282,12 @@ struct nes_cm_node { struct nes_timer_entry *send_entry; spinlock_t retrans_list_lock; - struct list_head recv_list; - spinlock_t recv_list_lock; + struct nes_timer_entry *recv_entry; int send_write0; union { struct ietf_mpa_frame mpa_frame; - u8 mpa_frame_buf[NES_CM_DEFAULT_MTU]; + u8 mpa_frame_buf[MAX_CM_BUFFER]; }; u16 mpa_frame_size; struct iw_cm_id *cm_id; @@ -326,6 +329,7 @@ enum nes_cm_event_type { NES_CM_EVENT_MPA_REQ, NES_CM_EVENT_MPA_CONNECT, NES_CM_EVENT_MPA_ACCEPT, + NES_CM_EVENT_MPA_REJECT, NES_CM_EVENT_MPA_ESTABLISHED, NES_CM_EVENT_CONNECTED, NES_CM_EVENT_CLOSED, -- 1.5.3.3 From faisal.latif at intel.com Thu Mar 5 12:48:29 2009 From: faisal.latif at intel.com (Faisal Latif) Date: Thu, 5 Mar 2009 14:48:29 -0600 Subject: [ofa-general] [PATCH] RDMA/nes: Make iw_nes non-LLTX driver Message-ID: <20090305204829.GA10488@flatif-MOBL> There is softirq kernel warning caused by skb_linearize() call from the iw_nes with IRQs disabled. By making it a non-LLTX, we do not disable the IRQs when skb_linearize() is called. Remove the sq_lock as it is not needed for non-LLTX. Fix the ethtool not to show the counter for sq_lock. Reported-by: aluno3 at poczta.onet.pl Signed-off-by: Faisal Latif --- drivers/infiniband/hw/nes/nes_hw.c | 5 +- drivers/infiniband/hw/nes/nes_hw.h | 2 - drivers/infiniband/hw/nes/nes_nic.c | 140 +++++++++++++++-------------------- 3 files changed, 63 insertions(+), 84 deletions(-) diff --git a/drivers/infiniband/hw/nes/nes_hw.c b/drivers/infiniband/hw/nes/nes_hw.c index 1c5e946..9a51f25 100644 --- a/drivers/infiniband/hw/nes/nes_hw.c +++ b/drivers/infiniband/hw/nes/nes_hw.c @@ -1644,7 +1644,6 @@ int nes_init_nic_qp(struct nes_device *nesdev, struct net_device *netdev) nesvnic->post_cqp_request = nes_post_cqp_request; nesvnic->mcrq_mcast_filter = NULL; - spin_lock_init(&nesvnic->nic.sq_lock); spin_lock_init(&nesvnic->nic.rq_lock); /* setup the RQ */ @@ -2632,9 +2631,9 @@ void nes_nic_ce_handler(struct nes_device *nesdev, struct nes_hw_nic_cq *cq) } else break; } - if (skb) - dev_kfree_skb_any(skb); } + if (skb) + dev_kfree_skb_any(skb); nesnic->sq_tail++; nesnic->sq_tail &= nesnic->sq_size-1; if (sq_cqes > 128) { diff --git a/drivers/infiniband/hw/nes/nes_hw.h b/drivers/infiniband/hw/nes/nes_hw.h index bf7ecfa..f41a871 100644 --- a/drivers/infiniband/hw/nes/nes_hw.h +++ b/drivers/infiniband/hw/nes/nes_hw.h @@ -876,7 +876,6 @@ struct nes_hw_nic { u8 replenishing_rq; u8 reserved; - spinlock_t sq_lock; spinlock_t rq_lock; }; @@ -1148,7 +1147,6 @@ struct nes_ib_device; struct nes_vnic { struct nes_ib_device *nesibdev; u64 sq_full; - u64 sq_locked; u64 tso_requests; u64 segmented_tso_requests; u64 linearized_skbs; diff --git a/drivers/infiniband/hw/nes/nes_nic.c b/drivers/infiniband/hw/nes/nes_nic.c index ebc7e7b..adf95aa 100644 --- a/drivers/infiniband/hw/nes/nes_nic.c +++ b/drivers/infiniband/hw/nes/nes_nic.c @@ -400,8 +400,7 @@ static int nes_nic_send(struct sk_buff *skb, struct net_device *netdev) if (skb_headlen(skb) == skb->len) { if (skb_headlen(skb) <= NES_FIRST_FRAG_SIZE) { nic_sqe->wqe_words[NES_NIC_SQ_WQE_LENGTH_2_1_IDX] = 0; - nesnic->tx_skb[nesnic->sq_head] = NULL; - dev_kfree_skb(skb); + nesnic->tx_skb[nesnic->sq_head] = skb; } } else { /* Deal with Fragments */ @@ -453,7 +452,6 @@ static int nes_netdev_start_xmit(struct sk_buff *skb, struct net_device *netdev) u32 wqe_count=1; u32 send_rc; struct iphdr *iph; - unsigned long flags; __le16 *wqe_fragment_length; u32 nr_frags; u32 original_first_length; @@ -480,13 +478,6 @@ static int nes_netdev_start_xmit(struct sk_buff *skb, struct net_device *netdev) if (netif_queue_stopped(netdev)) return NETDEV_TX_BUSY; - local_irq_save(flags); - if (!spin_trylock(&nesnic->sq_lock)) { - local_irq_restore(flags); - nesvnic->sq_locked++; - return NETDEV_TX_LOCKED; - } - /* Check if SQ is full */ if ((((nesnic->sq_tail+(nesnic->sq_size*2))-nesnic->sq_head) & (nesnic->sq_size - 1)) == 1) { if (!netif_queue_stopped(netdev)) { @@ -498,7 +489,6 @@ static int nes_netdev_start_xmit(struct sk_buff *skb, struct net_device *netdev) } } nesvnic->sq_full++; - spin_unlock_irqrestore(&nesnic->sq_lock, flags); return NETDEV_TX_BUSY; } @@ -531,7 +521,6 @@ sq_no_longer_full: } } nesvnic->sq_full++; - spin_unlock_irqrestore(&nesnic->sq_lock, flags); nes_debug(NES_DBG_NIC_TX, "%s: HNIC SQ full- TSO request has too many frags!\n", netdev->name); return NETDEV_TX_BUSY; @@ -656,17 +645,13 @@ tso_sq_no_longer_full: skb_set_transport_header(skb, hoffset); skb_set_network_header(skb, nhoffset); send_rc = nes_nic_send(skb, netdev); - if (send_rc != NETDEV_TX_OK) { - spin_unlock_irqrestore(&nesnic->sq_lock, flags); + if (send_rc != NETDEV_TX_OK) return NETDEV_TX_OK; - } } } else { send_rc = nes_nic_send(skb, netdev); - if (send_rc != NETDEV_TX_OK) { - spin_unlock_irqrestore(&nesnic->sq_lock, flags); + if (send_rc != NETDEV_TX_OK) return NETDEV_TX_OK; - } } barrier(); @@ -676,7 +661,6 @@ tso_sq_no_longer_full: (wqe_count << 24) | (1 << 23) | nesvnic->nic.qp_id); netdev->trans_start = jiffies; - spin_unlock_irqrestore(&nesnic->sq_lock, flags); return NETDEV_TX_OK; } @@ -1012,7 +996,6 @@ static const char nes_ethtool_stringset[][ETH_GSTRING_LEN] = { "Pause Frames Received", "Internal Routing Errors", "SQ SW Dropped SKBs", - "SQ Locked", "SQ Full", "Segmented TSO Requests", "Rx Symbol Errors", @@ -1129,16 +1112,17 @@ static void nes_netdev_get_ethtool_stats(struct net_device *netdev, struct nes_device *nesdev = nesvnic->nesdev; u32 nic_count; u32 u32temp; + u32 index = 0; target_ethtool_stats->n_stats = NES_ETHTOOL_STAT_COUNT; - target_stat_values[0] = nesvnic->nesdev->link_status_interrupts; - target_stat_values[1] = nesvnic->linearized_skbs; - target_stat_values[2] = nesvnic->tso_requests; + target_stat_values[index] = nesvnic->nesdev->link_status_interrupts; + target_stat_values[++index] = nesvnic->linearized_skbs; + target_stat_values[++index] = nesvnic->tso_requests; u32temp = nes_read_indexed(nesdev, NES_IDX_MAC_TX_PAUSE_FRAMES + (nesvnic->nesdev->mac_index*0x200)); nesvnic->nesdev->mac_pause_frames_sent += u32temp; - target_stat_values[3] = nesvnic->nesdev->mac_pause_frames_sent; + target_stat_values[++index] = nesvnic->nesdev->mac_pause_frames_sent; u32temp = nes_read_indexed(nesdev, NES_IDX_MAC_RX_PAUSE_FRAMES + (nesvnic->nesdev->mac_index*0x200)); @@ -1209,60 +1193,59 @@ static void nes_netdev_get_ethtool_stats(struct net_device *netdev, nesvnic->endnode_ipv4_tcp_retransmits += u32temp; } - target_stat_values[4] = nesvnic->nesdev->mac_pause_frames_received; - target_stat_values[5] = nesdev->nesadapter->nic_rx_eth_route_err; - target_stat_values[6] = nesvnic->tx_sw_dropped; - target_stat_values[7] = nesvnic->sq_locked; - target_stat_values[8] = nesvnic->sq_full; - target_stat_values[9] = nesvnic->segmented_tso_requests; - target_stat_values[10] = nesvnic->nesdev->mac_rx_symbol_err_frames; - target_stat_values[11] = nesvnic->nesdev->mac_rx_jabber_frames; - target_stat_values[12] = nesvnic->nesdev->mac_rx_oversized_frames; - target_stat_values[13] = nesvnic->nesdev->mac_rx_short_frames; - target_stat_values[14] = nesvnic->endnode_nstat_rx_discard; - target_stat_values[15] = nesvnic->endnode_nstat_rx_octets; - target_stat_values[16] = nesvnic->endnode_nstat_rx_frames; - target_stat_values[17] = nesvnic->endnode_nstat_tx_octets; - target_stat_values[18] = nesvnic->endnode_nstat_tx_frames; - target_stat_values[19] = mh_detected; - target_stat_values[20] = mh_pauses_sent; - target_stat_values[21] = nesvnic->endnode_ipv4_tcp_retransmits; - target_stat_values[22] = atomic_read(&cm_connects); - target_stat_values[23] = atomic_read(&cm_accepts); - target_stat_values[24] = atomic_read(&cm_disconnects); - target_stat_values[25] = atomic_read(&cm_connecteds); - target_stat_values[26] = atomic_read(&cm_connect_reqs); - target_stat_values[27] = atomic_read(&cm_rejects); - target_stat_values[28] = atomic_read(&mod_qp_timouts); - target_stat_values[29] = atomic_read(&qps_created); - target_stat_values[30] = atomic_read(&sw_qps_destroyed); - target_stat_values[31] = atomic_read(&qps_destroyed); - target_stat_values[32] = atomic_read(&cm_closes); - target_stat_values[33] = cm_packets_sent; - target_stat_values[34] = cm_packets_bounced; - target_stat_values[35] = cm_packets_created; - target_stat_values[36] = cm_packets_received; - target_stat_values[37] = cm_packets_dropped; - target_stat_values[38] = cm_packets_retrans; - target_stat_values[39] = cm_listens_created; - target_stat_values[40] = cm_listens_destroyed; - target_stat_values[41] = cm_backlog_drops; - target_stat_values[42] = atomic_read(&cm_loopbacks); - target_stat_values[43] = atomic_read(&cm_nodes_created); - target_stat_values[44] = atomic_read(&cm_nodes_destroyed); - target_stat_values[45] = atomic_read(&cm_accel_dropped_pkts); - target_stat_values[46] = atomic_read(&cm_resets_recvd); - target_stat_values[47] = int_mod_timer_init; - target_stat_values[48] = int_mod_cq_depth_1; - target_stat_values[49] = int_mod_cq_depth_4; - target_stat_values[50] = int_mod_cq_depth_16; - target_stat_values[51] = int_mod_cq_depth_24; - target_stat_values[52] = int_mod_cq_depth_32; - target_stat_values[53] = int_mod_cq_depth_128; - target_stat_values[54] = int_mod_cq_depth_256; - target_stat_values[55] = nesvnic->lro_mgr.stats.aggregated; - target_stat_values[56] = nesvnic->lro_mgr.stats.flushed; - target_stat_values[57] = nesvnic->lro_mgr.stats.no_desc; + target_stat_values[++index] = nesvnic->nesdev->mac_pause_frames_received; + target_stat_values[++index] = nesdev->nesadapter->nic_rx_eth_route_err; + target_stat_values[++index] = nesvnic->tx_sw_dropped; + target_stat_values[++index] = nesvnic->sq_full; + target_stat_values[++index] = nesvnic->segmented_tso_requests; + target_stat_values[++index] = nesvnic->nesdev->mac_rx_symbol_err_frames; + target_stat_values[++index] = nesvnic->nesdev->mac_rx_jabber_frames; + target_stat_values[++index] = nesvnic->nesdev->mac_rx_oversized_frames; + target_stat_values[++index] = nesvnic->nesdev->mac_rx_short_frames; + target_stat_values[++index] = nesvnic->endnode_nstat_rx_discard; + target_stat_values[++index] = nesvnic->endnode_nstat_rx_octets; + target_stat_values[++index] = nesvnic->endnode_nstat_rx_frames; + target_stat_values[++index] = nesvnic->endnode_nstat_tx_octets; + target_stat_values[++index] = nesvnic->endnode_nstat_tx_frames; + target_stat_values[++index] = mh_detected; + target_stat_values[++index] = mh_pauses_sent; + target_stat_values[++index] = nesvnic->endnode_ipv4_tcp_retransmits; + target_stat_values[++index] = atomic_read(&cm_connects); + target_stat_values[++index] = atomic_read(&cm_accepts); + target_stat_values[++index] = atomic_read(&cm_disconnects); + target_stat_values[++index] = atomic_read(&cm_connecteds); + target_stat_values[++index] = atomic_read(&cm_connect_reqs); + target_stat_values[++index] = atomic_read(&cm_rejects); + target_stat_values[++index] = atomic_read(&mod_qp_timouts); + target_stat_values[++index] = atomic_read(&qps_created); + target_stat_values[++index] = atomic_read(&sw_qps_destroyed); + target_stat_values[++index] = atomic_read(&qps_destroyed); + target_stat_values[++index] = atomic_read(&cm_closes); + target_stat_values[++index] = cm_packets_sent; + target_stat_values[++index] = cm_packets_bounced; + target_stat_values[++index] = cm_packets_created; + target_stat_values[++index] = cm_packets_received; + target_stat_values[++index] = cm_packets_dropped; + target_stat_values[++index] = cm_packets_retrans; + target_stat_values[++index] = cm_listens_created; + target_stat_values[++index] = cm_listens_destroyed; + target_stat_values[++index] = cm_backlog_drops; + target_stat_values[++index] = atomic_read(&cm_loopbacks); + target_stat_values[++index] = atomic_read(&cm_nodes_created); + target_stat_values[++index] = atomic_read(&cm_nodes_destroyed); + target_stat_values[++index] = atomic_read(&cm_accel_dropped_pkts); + target_stat_values[++index] = atomic_read(&cm_resets_recvd); + target_stat_values[++index] = int_mod_timer_init; + target_stat_values[++index] = int_mod_cq_depth_1; + target_stat_values[++index] = int_mod_cq_depth_4; + target_stat_values[++index] = int_mod_cq_depth_16; + target_stat_values[++index] = int_mod_cq_depth_24; + target_stat_values[++index] = int_mod_cq_depth_32; + target_stat_values[++index] = int_mod_cq_depth_128; + target_stat_values[++index] = int_mod_cq_depth_256; + target_stat_values[++index] = nesvnic->lro_mgr.stats.aggregated; + target_stat_values[++index] = nesvnic->lro_mgr.stats.flushed; + target_stat_values[++index] = nesvnic->lro_mgr.stats.no_desc; } @@ -1620,7 +1603,6 @@ struct net_device *nes_netdev_init(struct nes_device *nesdev, netif_napi_add(netdev, &nesvnic->napi, nes_netdev_poll, 128); nes_debug(NES_DBG_INIT, "Enabling VLAN Insert/Delete.\n"); netdev->features |= NETIF_F_HW_VLAN_TX | NETIF_F_HW_VLAN_RX; - netdev->features |= NETIF_F_LLTX; /* Fill in the port structure */ nesvnic->netdev = netdev; -- 1.5.3.3 From sashak at voltaire.com Thu Mar 5 15:55:50 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 01:55:50 +0200 Subject: [ofa-general] Re: [PATCHv2] Add pkey table support to osm_get_all_port_attrs In-Reply-To: <20090227184450.GA15668@comcast.net> References: <20090227184450.GA15668@comcast.net> Message-ID: <20090305235550.GB3784@sashak.voltaire.com> Hi Hal, On 13:44 Fri 27 Feb , Hal Rosenstock wrote: > > Only supported in osm_vendor_ibumad.c (separate patch for other > vendor layers) > Also, update applications using this (osmtest, opensm) > > Signed-off-by: Hal Rosenstock > > --- > Changes from v1: > Only copy number of pkeys indicated > Also, don't indicate insufficient memory error if insufficient pkey space > supplied and always return number of pkeys that the port supports > > Note: initialization prior to get_all_port_attrs call not changed > since it is faster this way > > Other patch for other vendor layers still appropriate following this > ibutils patch to come I think ibutils patch should go first - port_attr_array initialization will not hurt, but doing it in opposite order will leave ibutils in broken state. > > diff --git a/opensm/libvendor/osm_vendor_ibumad.c b/opensm/libvendor/osm_vendor_ibumad.c > index 734a860..7a578ea 100644 > --- a/opensm/libvendor/osm_vendor_ibumad.c > +++ b/opensm/libvendor/osm_vendor_ibumad.c > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2009 HNR Consulting. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -556,12 +557,13 @@ osm_vendor_get_all_port_attr(IN osm_vendor_t * const p_vend, > umad_ca_t ca; > ib_port_attr_t *attr = p_attr_array; > unsigned done = 0; > - int r, i, j; > + int r, i, j, k; > > OSM_LOG_ENTER(p_vend->p_log); > > CL_ASSERT(p_vend && p_num_ports); > > + r = 0; > if (!*p_num_ports) { > r = IB_INVALID_PARAMETER; > OSM_LOG(p_vend->p_log, OSM_LOG_ERROR, "ERR 5418: " > @@ -576,9 +578,7 @@ osm_vendor_get_all_port_attr(IN osm_vendor_t * const p_vend, > } > > for (i = 0; i < p_vend->ca_count && !done; i++) { > - /* > - * For each CA, retrieve the port guids > - */ > + /* For each CA, retrieve the port attributes */ > if (umad_get_ca(p_vend->ca_names[i], &ca) == 0) { > if (ca.node_type < 1 || ca.node_type > 3) > continue; > @@ -590,6 +590,12 @@ osm_vendor_get_all_port_attr(IN osm_vendor_t * const p_vend, > attr->port_num = ca.ports[j]->portnum; > attr->sm_lid = ca.ports[j]->sm_lid; > attr->link_state = ca.ports[j]->state; > + if (attr->num_pkeys && attr->p_pkey_table) { > + for (k = 0; k < attr->num_pkeys; k++) > + attr->p_pkey_table[k] = > + cl_hton16(ca.ports[j]->pkeys[k]); And when attr->num_pkeys > ca.ports[j]->pkeys_size? It will copy garbage? > + } > + attr->num_pkeys = ca.ports[j]->pkeys_size; > attr++; > if (attr - p_attr_array > *p_num_ports) { > done = 1; > @@ -601,7 +607,6 @@ osm_vendor_get_all_port_attr(IN osm_vendor_t * const p_vend, > } > > *p_num_ports = attr - p_attr_array; > - r = 0; > > Exit: > OSM_LOG_EXIT(p_vend->p_log); > diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c > index 47fd658..1507fff 100644 > --- a/opensm/opensm/main.c > +++ b/opensm/opensm/main.c > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2009 HNR Consulting. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU There should be initialization? > diff --git a/opensm/osmtest/main.c b/opensm/osmtest/main.c > index f87e33b..bc8999d 100644 Sasha From sashak at voltaire.com Thu Mar 5 15:59:03 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 01:59:03 +0200 Subject: [ofa-general] Re: [PATCH] opensm/osm_perfmgr.c: In osm_perfmgr_shutdown, add missing cl_disp_unregister In-Reply-To: <20090227184557.GB15668@comcast.net> References: <20090227184557.GB15668@comcast.net> Message-ID: <20090305235855.GC3784@sashak.voltaire.com> On 13:45 Fri 27 Feb , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Thu Mar 5 16:00:57 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 02:00:57 +0200 Subject: [ofa-general] Re: [PATCH] opensm/osm_perfmgr.c: Improve assert in osm_pc_rcv_process In-Reply-To: <20090227211851.GA25061@comcast.net> References: <20090227211851.GA25061@comcast.net> Message-ID: <20090306000057.GD3784@sashak.voltaire.com> On 16:18 Fri 27 Feb , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From hal.rosenstock at gmail.com Thu Mar 5 18:00:23 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 5 Mar 2009 21:00:23 -0500 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCHv2] Add pkey table support to osm_get_all_port_attrs In-Reply-To: <20090305235550.GB3784@sashak.voltaire.com> References: <20090227184450.GA15668@comcast.net> <20090305235550.GB3784@sashak.voltaire.com> Message-ID: Sasha, On Thu, Mar 5, 2009 at 6:55 PM, Sasha Khapyorsky wrote: > Hi Hal, > > On 13:44 Fri 27 Feb     , Hal Rosenstock wrote: >> >> Only supported in osm_vendor_ibumad.c (separate patch for other >> vendor layers) >> Also, update applications using this (osmtest, opensm) >> >> Signed-off-by: Hal Rosenstock >> >> --- >> Changes from v1: >> Only copy number of pkeys indicated >> Also, don't indicate insufficient memory error if insufficient pkey space >> supplied and always return number of pkeys that the port supports >> >> Note: initialization prior to get_all_port_attrs call not changed >> since it is faster this way >> >> Other patch for other vendor layers still appropriate following this >> ibutils patch to come > > I think ibutils patch should go first - port_attr_array initialization > will not hurt, but doing it in opposite order will leave ibutils in > broken state. Fine; I'll shortly send this patch but hope the rest won't be pended waiting for it to be applied. > - Show quoted text - >> >> diff --git a/opensm/libvendor/osm_vendor_ibumad.c b/opensm/libvendor/osm_vendor_ibumad.c >> index 734a860..7a578ea 100644 >> --- a/opensm/libvendor/osm_vendor_ibumad.c >> +++ b/opensm/libvendor/osm_vendor_ibumad.c >> @@ -2,6 +2,7 @@ >>   * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. >>   * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. >>   * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. >> + * Copyright (c) 2009 HNR Consulting. All rights reserved. >>   * >>   * This software is available to you under a choice of one of two >>   * licenses.  You may choose to be licensed under the terms of the GNU >> @@ -556,12 +557,13 @@ osm_vendor_get_all_port_attr(IN osm_vendor_t * const p_vend, >>       umad_ca_t ca; >>       ib_port_attr_t *attr = p_attr_array; >>       unsigned done = 0; >> -     int r, i, j; >> +     int r, i, j, k; >> >>       OSM_LOG_ENTER(p_vend->p_log); >> >>       CL_ASSERT(p_vend && p_num_ports); >> >> +     r = 0; >>       if (!*p_num_ports) { >>               r = IB_INVALID_PARAMETER; >>               OSM_LOG(p_vend->p_log, OSM_LOG_ERROR, "ERR 5418: " >> @@ -576,9 +578,7 @@ osm_vendor_get_all_port_attr(IN osm_vendor_t * const p_vend, >>       } >> >>       for (i = 0; i < p_vend->ca_count && !done; i++) { >> -             /* >> -              * For each CA, retrieve the port guids >> -              */ >> +             /* For each CA, retrieve the port attributes */ >>               if (umad_get_ca(p_vend->ca_names[i], &ca) == 0) { >>                       if (ca.node_type < 1 || ca.node_type > 3) >>                               continue; >> @@ -590,6 +590,12 @@ osm_vendor_get_all_port_attr(IN osm_vendor_t * const p_vend, >>                               attr->port_num = ca.ports[j]->portnum; >>                               attr->sm_lid = ca.ports[j]->sm_lid; >>                               attr->link_state = ca.ports[j]->state; >> +                             if (attr->num_pkeys && attr->p_pkey_table) { >> +                                     for (k = 0; k < attr->num_pkeys; k++) >> +                                             attr->p_pkey_table[k] = >> +                                                     cl_hton16(ca.ports[j]->pkeys[k]); > > And when attr->num_pkeys > ca.ports[j]->pkeys_size? It will copy > garbage? I'll fix this in v3 of this patch once I understand what you mean below. >> +                             } >> +                             attr->num_pkeys = ca.ports[j]->pkeys_size; >>                               attr++; >>                               if (attr - p_attr_array > *p_num_ports) { >>                                       done = 1; >> @@ -601,7 +607,6 @@ osm_vendor_get_all_port_attr(IN osm_vendor_t * const p_vend, >>       } >> >>       *p_num_ports = attr - p_attr_array; >> -     r = 0; >> >>  Exit: >>       OSM_LOG_EXIT(p_vend->p_log); >> diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c >> index 47fd658..1507fff 100644 >> --- a/opensm/opensm/main.c >> +++ b/opensm/opensm/main.c >> @@ -2,6 +2,7 @@ >>   * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. >>   * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. >>   * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. >> + * Copyright (c) 2009 HNR Consulting. All rights reserved. >>   * >>   * This software is available to you under a choice of one of two >>   * licenses.  You may choose to be licensed under the terms of the GNU > > There should be initialization? I don't understand your question. -- Hal >> diff --git a/opensm/osmtest/main.c b/opensm/osmtest/main.c >> index f87e33b..bc8999d 100644 > > Sasha > - Show quoted text - > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hnrose at comcast.net Thu Mar 5 18:16:46 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 5 Mar 2009 21:16:46 -0500 Subject: [ofa-general] [PATCH] ibutils/ibis: Prepare for pkeys support in osm_get_all_port_attr Message-ID: <20090306021646.GA3934@comcast.net> In order not to break tree, this initialization needs to preceed the change to the management tree where osm_get_all_port_attr (at least the umad vendor version) will support pkeys. Signed-off-by: Hal Rosenstock --- diff --git a/ibis/src/ibis.c b/ibis/src/ibis.c index b5662c6..7797719 100644 --- a/ibis/src/ibis.c +++ b/ibis/src/ibis.c @@ -196,11 +196,16 @@ ibis_get_ports_status( { uint32_t i; ib_api_status_t status; - ib_port_attr_t attr_array[GUID_ARRAY_SIZE]; + ib_port_attr_t attr_array[MAX_LOCAL_IBPORTS]; OSM_LOG_ENTER(&(IbisObj.log)); - *num_ports = GUID_ARRAY_SIZE; + for (i = 0; i < MAX_LOCAL_IBPORTS; i++) + { + attr_array[i].num_pkeys = 0; + attr_array[i].p_pkey_table = NULL; + } + *num_ports = MAX_LOCAL_IBPORTS; status = osm_vendor_get_all_port_attr( IbisObj.p_vendor, attr_array, diff --git a/ibis/src/ibis.i b/ibis/src/ibis.i index 1bc3f34..a153921 100644 --- a/ibis/src/ibis.i +++ b/ibis/src/ibis.i @@ -381,8 +381,8 @@ ibisp_is_debug(void) ibis_t *p_ibis = &IbisObj; uint32_t i; ib_api_status_t status; - uint32_t num_ports = GUID_ARRAY_SIZE; - ib_port_attr_t attr_array[GUID_ARRAY_SIZE]; + uint32_t num_ports = MAX_LOCAL_IBPORTS; + ib_port_attr_t attr_array[MAX_LOCAL_IBPORTS]; static char res[128]; Tcl_Obj *p_obj; @@ -402,6 +402,11 @@ ibisp_is_debug(void) return TCL_ERROR; } + for (i = 0; i < num_ports; i++) + { + attr_array[i].num_pkeys = 0; + attr_array[i].p_pkey_table = NULL; + } /* Call the transport layer for a list of local port GUID values. diff --git a/ibis/src/ibis_base.h b/ibis/src/ibis_base.h index 654ed17..ad26148 100644 --- a/ibis/src/ibis_base.h +++ b/ibis/src/ibis_base.h @@ -65,7 +65,7 @@ BEGIN_C_DECLS #define IBIS_FILE_PATH_MAX PATH_MAX #endif -#define GUID_ARRAY_SIZE 32 +#define MAX_LOCAL_IBPORTS 32 #define DISP_HANDLE_MAX 32 END_C_DECLS diff --git a/ibis/src/ibis_wrap.c b/ibis/src/ibis_wrap.c index f4d5b64..b3fc659 100644 --- a/ibis/src/ibis_wrap.c +++ b/ibis/src/ibis_wrap.c @@ -3134,8 +3134,8 @@ typedef struct { ibis_t *p_ibis = &IbisObj; uint32_t i; ib_api_status_t status; - uint32_t num_ports = GUID_ARRAY_SIZE; - ib_port_attr_t attr_array[GUID_ARRAY_SIZE]; + uint32_t num_ports = MAX_LOCAL_IBPORTS; + ib_port_attr_t attr_array[MAX_LOCAL_IBPORTS]; static char res[128]; Tcl_Obj *p_obj; @@ -3155,6 +3155,11 @@ typedef struct { return TCL_ERROR; } + for (i = 0; i < MAX_LOCAL_IBPORTS; i++) + { + attr_array[i].num_pkeys = 0; + attr_array[i].p_pkey_table = NULL; + } /* Call the transport layer for a list of local port GUID values. diff --git a/ibis/src/ibissh_wrap.cpp b/ibis/src/ibissh_wrap.cpp index e1b43b7..2e498c7 100644 --- a/ibis/src/ibissh_wrap.cpp +++ b/ibis/src/ibissh_wrap.cpp @@ -3134,8 +3134,8 @@ typedef struct { ibis_t *p_ibis = &IbisObj; uint32_t i; ib_api_status_t status; - uint32_t num_ports = GUID_ARRAY_SIZE; - ib_port_attr_t attr_array[GUID_ARRAY_SIZE]; + uint32_t num_ports = MAX_LOCAL_IBPORTS; + ib_port_attr_t attr_array[MAX_LOCAL_IBPORTS]; static char res[128]; Tcl_Obj *p_obj; @@ -3155,6 +3155,11 @@ typedef struct { return TCL_ERROR; } + for (i = 0; i < num_ports; i++) + { + attr_array[i].num_pkeys = 0; + attr_array[i].p_pkey_table = NULL; + } /* Call the transport layer for a list of local port GUID values. From poknam at gmail.com Thu Mar 5 18:57:45 2009 From: poknam at gmail.com (PN) Date: Fri, 6 Mar 2009 10:57:45 +0800 Subject: ***SPAM*** Re: [ofa-general] OFED 1.4.1 is available In-Reply-To: <5D49E7A8952DC44FB38C38FA0D758EAD01F50B50@mtlexch01.mtl.com> References: <5D49E7A8952DC44FB38C38FA0D758EAD01F50B50@mtlexch01.mtl.com> Message-ID: <92daa7bf0903051857v5754f52fjc0b2b10324ddd4ac@mail.gmail.com> Dear all, Sorry, I have another problem on installing the OFED-1.4.1-rc1 on CentOS 5.2. I can build all the rpm, but there is an error when installing the mpitests_mvapich2_intel-3.1-891.x86_64.rpm .... Install mpitests_mvapich_gcc RPM: Running rpm -iv /opt/software/packages/ofed/OFED-1.4.1-rc1/OFED-1.4.1-rc1/RPMS/centos-release-5-2.el5.centos/x86_64/mpitests_mvapich_gcc-3.1-891.x86_64.rpm Install mpitests_mvapich_intel RPM: Running rpm -iv /opt/software/packages/ofed/OFED-1.4.1-rc1/OFED-1.4.1-rc1/RPMS/centos-release-5-2.el5.centos/x86_64/mpitests_mvapich_intel-3.1-891.x86_64.rpm Install mpitests_mvapich2_gcc RPM: Running rpm -iv /opt/software/packages/ofed/OFED-1.4.1-rc1/OFED-1.4.1-rc1/RPMS/centos-release-5-2.el5.centos/x86_64/mpitests_mvapich2_gcc-3.1-891.x86_64.rpm Install mpitests_mvapich2_intel RPM: Running rpm -iv /opt/software/packages/ofed/OFED-1.4.1-rc1/OFED-1.4.1-rc1/RPMS/centos-release-5-2.el5.centos/x86_64/mpitests_mvapich2_intel-3.1-891.x86_64.rpm Failed to install mpitests_mvapich2_intel RPM See /tmp/OFED.29515.logs/mpitests_mvapich2_intel.rpminstall.log # cat /tmp/OFED.29515.logs/mpitests_mvapich2_intel.rpminstall.log error: Failed dependencies: libimf.so()(64bit) is needed by mpitests_mvapich2_intel-3.1-891.x86_64 libintlc.so.5()(64bit) is needed by mpitests_mvapich2_intel-3.1-891.x86_64 libsvml.so()(64bit) is needed by mpitests_mvapich2_intel-3.1-891.x86_64 I've installed Intel Compiler 11.0.081 and I found that all the required .so are located in /opt/intel/Compiler/11.0/081/lib/intel64 and the location is in LD_LIBRARY_PATH. # echo $LD_LIBRARY_PATH /opt/intel/Compiler/11.0/081/lib/intel64:/opt/intel/Compiler/11.0/081/ipp/em64t/sharedlib:/opt/intel/Compiler/11.0/081/mkl/lib/em64t:/opt/intel/Compiler/11.0/081/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib:/opt/intel/Compiler/11.0/081/lib/intel64:/opt/intel/Compiler/11.0/081/ipp/em64t/sharedlib:/opt/intel/Compiler/11.0/081/mkl/lib/em64t:/opt/intel/Compiler/11.0/081/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib Any idea on how to solve this? Thanks a lot. Regards, PN -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Thu Mar 5 21:50:00 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 07:50:00 +0200 Subject: [ofa-general] Re: [PATCH] opensm: Return error status when cl_disp_register fails In-Reply-To: <20090302120306.GA6493@comcast.net> References: <20090302120306.GA6493@comcast.net> Message-ID: <20090306055000.GE3784@sashak.voltaire.com> On 07:03 Mon 02 Mar , Hal Rosenstock wrote: > > Also, in perfmgr, return error on db_construct failure > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Thu Mar 5 21:53:35 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 07:53:35 +0200 Subject: [ofa-general] Re: [PATCH] opensm/ib_types.h: Add attribute ID for PortCountersExtended In-Reply-To: <20090302120357.GB6493@comcast.net> References: <20090302120357.GB6493@comcast.net> Message-ID: <20090306055328.GF3784@sashak.voltaire.com> On 07:03 Mon 02 Mar , Hal Rosenstock wrote: > > Also, add some PerfMgt ClassPortInfo CapabilityMask bit definitions > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Thu Mar 5 21:58:42 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 07:58:42 +0200 Subject: [ofa-general] Re: [PATCHv2] Add pkey table support to osm_get_all_port_attrs In-Reply-To: References: <20090227184450.GA15668@comcast.net> <20090305235550.GB3784@sashak.voltaire.com> Message-ID: <20090306055842.GG3784@sashak.voltaire.com> On 21:00 Thu 05 Mar , Hal Rosenstock wrote: > >> diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c > >> index 47fd658..1507fff 100644 > >> --- a/opensm/opensm/main.c > >> +++ b/opensm/opensm/main.c > >> @@ -2,6 +2,7 @@ > >> ?? * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > >> ?? * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. > >> ?? * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > >> + * Copyright (c) 2009 HNR Consulting. All rights reserved. > >> ?? * > >> ?? * This software is available to you under a choice of one of two > >> ?? * licenses. ??You may choose to be licensed under the terms of the GNU > > > > There should be initialization? > > I don't understand your question. There is no any change in this file (except Copyright). And I suppose there should be attr_array initialization in get_port_guid(). No? Sasha > > -- Hal > > >> diff --git a/opensm/osmtest/main.c b/opensm/osmtest/main.c > >> index f87e33b..bc8999d 100644 > > > > Sasha > > - Show quoted text - > > _______________________________________________ > > general mailing list > > general at lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > From rdreier at cisco.com Thu Mar 5 21:56:30 2009 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 05 Mar 2009 21:56:30 -0800 Subject: [ofa-general][PATCH 1/2] mlx4_core: Link sensing support In-Reply-To: <49B02A6C.1000805@mellanox.co.il> (Yevgeny Petrilin's message of "Thu, 05 Mar 2009 21:39:24 +0200") References: <49AD25CE.8010208@mellanox.co.il> <49B02A6C.1000805@mellanox.co.il> Message-ID: > > > + queue_delayed_work(sense->sense_wq , &sense->sense_poll, > > > + round_jiffies(MLX4_SENSE_RANGE)); > > should be round_jiffies_relative, right? > I did not find critical difference between the two: Not sure what you mean by "critical" -- the difference is that one is correct and one is not; since queue_delayed_work() takes a relative timeout rather than an absolute time in jiffies, then round_jiffies_relative() should be used. > > Do we really another work queue, or can we share one queue for the > > catastrophic error and port sensing work? > There is no special reason for having another queue other then convenience. > The catastrophic work queue is defined in catas.c Just make a common thread for both in the most convenient spot and use it. We waste too many kernel threads as it is. - R. From rdreier at cisco.com Thu Mar 5 21:57:42 2009 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 05 Mar 2009 21:57:42 -0800 Subject: [ofa-general][PATCH 2/2] mlx4_core: Don't perform SET_PORT command for Eth port from core. In-Reply-To: <49B021BD.2060805@mellanox.co.il> (Yevgeny Petrilin's message of "Thu, 05 Mar 2009 21:02:21 +0200") References: <49AD25DE.9010501@mellanox.co.il> <49B021BD.2060805@mellanox.co.il> Message-ID: > The existing code (performing SET_PORT for Ethernet port) is not a > bug, bu it is unnecessary. It turns into a bug if patch 1/2 is > applied. So if someone doing a bisection lands between 1/2 and 2/2 then mlx4 is broken? Can the order of patches be reversed so that we avoid having a broken intermediate state? Or should this just be one big patch? From sashak at voltaire.com Thu Mar 5 23:00:08 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 09:00:08 +0200 Subject: [ofa-general] Re: [PATCH] opensm/console: Fixed osm_console poll to so it won't eat up all CPU when stdin is a pipe In-Reply-To: <49ABF834.90106@ext.bull.net> References: <49ABF834.90106@ext.bull.net> Message-ID: <20090306070008.GH3784@sashak.voltaire.com> Hi Nicolas, On 16:16 Mon 02 Mar , Nicolas Morey Chaisemartin wrote: > > Signed-off-by: Nicolas Morey-Chaisemartin > --- I guess that below is commit message, right? Assuming so, it should be before '---' line. > When opensm is called after a pipe to execute a command in the local console, it uses 100% CPU as osm_console poll returns as if there was something to read from stdin where there isn't. As far as I understand described usage (echo blah-blah | opensm ....) there should be nothing to read after 'echo' finished its job and poll() will set POLLHUP revents bit. > This patch fixes by closing the local console when the end of osm_console function has been reached, meaning poll returned a positive value but actually nothing to read. osm_console will return an error code which will unset the console_init_flag so it won't be reopened. > > For socket, the osm_console_close is called as it means the socket isn't in a proper state. And then what osm_console() will do? Wouldn't it run over fd_in? Sasha > > > opensm/include/opensm/osm_console.h | 2 +- > opensm/opensm/main.c | 6 ++++-- > opensm/opensm/osm_console.c | 19 +++++++++++++++---- > 3 files changed, 20 insertions(+), 7 deletions(-) > > diff --git a/opensm/include/opensm/osm_console.h b/opensm/include/opensm/osm_console.h > index 3ea8fa5..dcce1e4 100644 > --- a/opensm/include/opensm/osm_console.h > +++ b/opensm/include/opensm/osm_console.h > @@ -46,6 +46,6 @@ > > BEGIN_C_DECLS > // TODO replace p_osm > -void osm_console(osm_opensm_t * p_osm); > +int osm_console(osm_opensm_t * p_osm); > END_C_DECLS > #endif /* _OSM_CONSOLE_H_ */ > diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c > index 2271634..5a39f84 100644 > --- a/opensm/opensm/main.c > +++ b/opensm/opensm/main.c > @@ -494,8 +494,10 @@ int osm_manager_loop(osm_subn_opt_t * p_opt, osm_opensm_t * p_osm) > Sit here forever - dwell or do console i/o & cmds > */ > while (!osm_exit_flag) { > - if (console_init_flag) > - osm_console(p_osm); > + if (console_init_flag){ > + if(osm_console(p_osm)) > + console_init_flag = 0; > + } > else > cl_thread_suspend(10000); > > diff --git a/opensm/opensm/osm_console.c b/opensm/opensm/osm_console.c > index 63c5ea8..b6a8689 100644 > --- a/opensm/opensm/osm_console.c > +++ b/opensm/opensm/osm_console.c > @@ -1376,7 +1376,7 @@ static void parse_cmd_line(char *line, osm_opensm_t * p_osm) > } > } > > -void osm_console(osm_opensm_t * p_osm) > +int osm_console(osm_opensm_t * p_osm) > { > struct pollfd pollfd[2]; > char *p_line; > @@ -1409,7 +1409,7 @@ void osm_console(osm_opensm_t * p_osm) > } > > if (poll(fds, nfds, 1000) <= 0) > - return; > + return 0; > > #ifdef ENABLE_OSM_CONSOLE_SOCKET > if (pollfd[0].revents & POLLIN) { > @@ -1422,7 +1422,7 @@ void osm_console(osm_opensm_t * p_osm) > "ERR 4B04: Failed to accept console socket: %s\n", > strerror(errno)); > p_oct->in_fd = -1; > - return; > + return 0; > } > if (inet_ntop > (AF_INET, &sin.sin_addr, p_oct->client_ip, > @@ -1444,7 +1444,7 @@ void osm_console(osm_opensm_t * p_osm) > p_oct->client_hn, p_oct->client_ip); > close(new_fd); > } > - return; > + return 0; > } > #endif > > @@ -1462,5 +1462,16 @@ void osm_console(osm_opensm_t * p_osm) > osm_console_exit(p_oct, p_log); > if (p_line) > free(p_line); > + return 0; > } > + /* If we are here, there has been a problem with poll > + * so we exit the console not to eat up all the CPU > + * if it was a local console > + * If it was a socket, we just close it. > + */ > + if( p_oct->socket < 0 ) > + return -1; > + > + osm_console_exit(p_oct, p_log); > + return 0; > } > -- > 1.5.6.3 > From sashak at voltaire.com Thu Mar 5 23:17:04 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 09:17:04 +0200 Subject: [ofa-general] Re: [PATCH] libibumad/umad.c: Cosmetic changes In-Reply-To: <20090302210029.GA7285@comcast.net> References: <20090302210029.GA7285@comcast.net> Message-ID: <20090306071657.GI3784@sashak.voltaire.com> On 16:00 Mon 02 Mar , Hal Rosenstock wrote: > > In get_port, change variable name for better readability > In resolve_ca_name, use define rather than constant in umad_get_cas_names call > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Thu Mar 5 23:17:23 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 09:17:23 +0200 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_port.h: Fix a commentary typo In-Reply-To: <20090302210125.GB7285@comcast.net> References: <20090302210125.GB7285@comcast.net> Message-ID: <20090306071723.GJ3784@sashak.voltaire.com> On 16:01 Mon 02 Mar , Hal Rosenstock wrote: > > Also, cosmetic formatting change > > Signed-off-by: Hal Rosenstock Apllied. Thanks. Sasha From sashak at voltaire.com Thu Mar 5 23:17:52 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 09:17:52 +0200 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_state_mgr.c: Cosmetic commentary change In-Reply-To: <20090302210229.GC7285@comcast.net> References: <20090302210229.GC7285@comcast.net> Message-ID: <20090306071752.GK3784@sashak.voltaire.com> On 16:02 Mon 02 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Thu Mar 5 23:49:53 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 09:49:53 +0200 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_trap_rcv.c: Cosmetic changes In-Reply-To: <20090303152508.GA4425@comcast.net> References: <20090303152508.GA4425@comcast.net> Message-ID: <20090306074953.GA6258@sashak.voltaire.com> On 10:25 Tue 03 Mar , Hal Rosenstock wrote: > > Removed extraneous parentheses > Eliminated unneeded __ and __osm_ prefixes to static routine names > Fixed some commentary typos > > Signed-off-by: Hal Rosenstock Applied (with small change noted below). Thanks. > @@ -173,7 +173,7 @@ static uint32_t __osm_trap_calc_crc32(void *buffer, uint32_t count) > crc = i; > for (j = 8; j > 0; j--) > if (crc & 1) > - crc = (crc >> 1) ^ CRC32_POLYNOMIAL; > + crc = crc >> 1 ^ CRC32_POLYNOMIAL; I leaved it as it was - look slightly clearer with parentheses (at least for me). Sasha From sashak at voltaire.com Thu Mar 5 23:51:02 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 09:51:02 +0200 Subject: [ofa-general] Re: [PATCH] opensm/osm_ucast_ftree.c: Removed useless initialisation on switch indexes In-Reply-To: <49AD523C.2010501@ext.bull.net> References: <49AD523C.2010501@ext.bull.net> Message-ID: <20090306075102.GB6258@sashak.voltaire.com> On 16:52 Tue 03 Mar , Nicolas Morey Chaisemartin wrote: > > Signed-off-by: Nicolas Morey-Chaisemartin Applied. Thanks. Sasha From sashak at voltaire.com Thu Mar 5 23:52:14 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 09:52:14 +0200 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/libvendor/osm_vendor_ibumad.c: Commentary changes In-Reply-To: <20090303190304.GA14633@comcast.net> References: <20090303190304.GA14633@comcast.net> Message-ID: <20090306075214.GC6258@sashak.voltaire.com> On 14:03 Tue 03 Mar , Hal Rosenstock wrote: > > Also, formatting change > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Mar 6 00:06:24 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 10:06:24 +0200 Subject: [ofa-general] Re: [PATCH 1/10] libibmad: Clean up "new" interface In-Reply-To: <20090302095459.7e6d4a34.weiny2@llnl.gov> References: <20090219190525.322681b8.weiny2@llnl.gov> <20090301070013.GF3936@sashak.voltaire.com> <20090302095459.7e6d4a34.weiny2@llnl.gov> Message-ID: <20090306080624.GD6258@sashak.voltaire.com> Hi Ira, On 09:54 Mon 02 Mar , Ira Weiny wrote: > > > > > Create new mad_rpc_portid(struct ibmad_port *srcport) function > > > which mirrors madrpc_portid(void) > > > Mark all "old" functions with __attribute__ ((deprecated)) > > > > This generates a lot of warnings right now (even after all patch series > > applying it still have deprecated usages in libibmad itself). And this > > is not very good. I think our flow should have opposite direction - first > > to convert, then mark deprecated functions. > > > > Now as fast workaround I can mask depreciation by macro: > > > > #define DEPRECATED /* __attribute__ ((deprecated)) */ > > > > , and we will uncomment this when everything in tree will be converted. > > What if we put this #def in the internal *.c files only? I think it is simpler to convert them too :). Actually what I meant is something else - the problem is that after applying first patch in series we are getting a lot of warnings and this leaves some history states in this half-broken state (warnings, not an errors, but anyway). Which is not excellent by itself, but also complicates using some debugging things like 'git bisect'. So I wanted to mask deprecation with macro. > Then it will give a > warning to users but not on our internal code? I just want to give potential > users of the lib time and notice to change their code. I understand this - by uncommenting DEPRECATED macro we can get all those usages. > > Also after looking over patch series I see that all "original" function > > names become deprecated and replaces by its *_via() brothers. How do > > you see the next step? Will we remove old names and have almost all API > > calls with useless then _via suffix? > > Unless you want to break the API I don't see a way around this. "_via" or > some other "useless" name change will have to be used? I simply used "_via" > because it was already there. > > I don't know of many people using the lib so if you want we could just change > the "original" functions, break the API, and bump the library version. If you > don't think there are many users this would be cleaner going forward. But I > was trying to move more slowly than that. Ok, let's move slowly... When the conversion will be ready we could rename the functions to its original names and bump library version. Sasha From nicolas.morey-chaisemartin at ext.bull.net Fri Mar 6 00:27:45 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Fri, 06 Mar 2009 09:27:45 +0100 Subject: [ofa-general] Re: [PATCH] opensm/console: Fixed osm_console poll to so it won't eat up all CPU when stdin is a pipe In-Reply-To: <20090306070008.GH3784@sashak.voltaire.com> References: <49ABF834.90106@ext.bull.net> <20090306070008.GH3784@sashak.voltaire.com> Message-ID: <49B0DE81.4020407@ext.bull.net> Sasha Khapyorsky wrote: > Hi Nicolas, > > On 16:16 Mon 02 Mar , Nicolas Morey Chaisemartin wrote: >> Signed-off-by: Nicolas Morey-Chaisemartin >> --- > > I guess that below is commit message, right? Assuming so, it should be > before '---' line. > >> When opensm is called after a pipe to execute a command in the local console, it uses 100% CPU as osm_console poll returns as if there was something to read from stdin where there isn't. > > As far as I understand described usage (echo blah-blah | opensm ....) > there should be nothing to read after 'echo' finished its job and poll() > will set POLLHUP revents bit. > You're right. Although my man page says it's for output only, it works. >> This patch fixes by closing the local console when the end of osm_console function has been reached, meaning poll returned a positive value but actually nothing to read. osm_console will return an error code which will unset the console_init_flag so it won't be reopened. >> >> For socket, the osm_console_close is called as it means the socket isn't in a proper state. > > And then what osm_console() will do? Wouldn't it run over fd_in? > Well, maybe it could just kill the current connection (cio_close). By the way shouldn't the quit_parse function call cio_close instead od osm_console_close. In the current implementation, calling exit kills the socket so there can be no further connection. Nicolas From sashak at voltaire.com Fri Mar 6 00:45:16 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 10:45:16 +0200 Subject: [ofa-general] Re: [PATCH] opensm/console: Fixed osm_console poll to so it won't eat up all CPU when stdin is a pipe In-Reply-To: <49B0DE81.4020407@ext.bull.net> References: <49ABF834.90106@ext.bull.net> <20090306070008.GH3784@sashak.voltaire.com> <49B0DE81.4020407@ext.bull.net> Message-ID: <20090306084510.GE6258@sashak.voltaire.com> On 09:27 Fri 06 Mar , Nicolas Morey Chaisemartin wrote: > >> > >> For socket, the osm_console_close is called as it means the socket isn't in a proper state. > > > > And then what osm_console() will do? Wouldn't it run over fd_in? > > > Well, maybe it could just kill the current connection (cio_close). > > By the way shouldn't the quit_parse function call cio_close instead od osm_console_close. Yes, this seems reasonable. But it is another patch please. Sasha From sashak at voltaire.com Fri Mar 6 00:50:18 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 10:50:18 +0200 Subject: [ofa-general] Re: [PATCH 2/10] infiniband-diags: Convert ibaddr to "new" ibmad interface In-Reply-To: <20090302160659.0807fb24.weiny2@llnl.gov> References: <20090219190528.11c080f8.weiny2@llnl.gov> <20090301080804.GH3936@sashak.voltaire.com> <20090302160659.0807fb24.weiny2@llnl.gov> Message-ID: <20090306085002.GF6258@sashak.voltaire.com> On 16:06 Mon 02 Mar , Ira Weiny wrote: > > > diff --git a/infiniband-diags/src/ibdiag_common.c b/infiniband-diags/src/ibdiag_common.c > > > index 5f2472d..609df69 100644 > > > --- a/infiniband-diags/src/ibdiag_common.c > > > +++ b/infiniband-diags/src/ibdiag_common.c > > > @@ -179,7 +179,8 @@ static int process_opt(int ch, char *optarg) > > > ibd_timeout = val; > > > break; > > > case 's': > > > - if (ib_resolve_portid_str(&sm_portid, optarg, IB_DEST_LID, 0) < 0) > > > + if (ib_resolve_portid_str_via(&sm_portid, optarg, IB_DEST_LID, > > > + 0, NULL) < 0) > > > > This may be not correct (not introduced by those patches) - then host is > > connected to multiple subnets we need to resolve SM LID on requested > > one, eg. using srcport... > > > > Would you accept adding srcport to ibdiag_process_opts? Or would you prefer > something more generic? More generic would be better :) BTW it could be not so easy - when ibdiag_process_opts() is running srcport is not initialized yet. Sasha From sashak at voltaire.com Fri Mar 6 01:01:13 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 11:01:13 +0200 Subject: [ofa-general] Re: [PATCH 1/10] libibmad: Clean up "new" interface In-Reply-To: <20090306080624.GD6258@sashak.voltaire.com> References: <20090219190525.322681b8.weiny2@llnl.gov> <20090301070013.GF3936@sashak.voltaire.com> <20090302095459.7e6d4a34.weiny2@llnl.gov> <20090306080624.GD6258@sashak.voltaire.com> Message-ID: <20090306085350.GG6258@sashak.voltaire.com> On 10:06 Fri 06 Mar , Sasha Khapyorsky wrote: > > Actually what I meant is something else - the problem is that after > applying first patch in series we are getting a lot of warnings and this > leaves some history states in this half-broken state (warnings, not > an errors, but anyway). Which is not excellent by itself, but also > complicates using some debugging things like 'git bisect'. So I wanted to > mask deprecation with macro. I'm applying this patch with deprecation masked by macro. So we will be able to move forward with other things. Waiting for updated infiniband-diags patch(es). Sasha From nicolas.morey-chaisemartin at ext.bull.net Fri Mar 6 01:17:42 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Fri, 06 Mar 2009 10:17:42 +0100 Subject: [ofa-general] [PATCH] opensm/console: Fixed osm_console poll to handle POLLHUP Message-ID: <49B0EA36.9040609@ext.bull.net> When opensm is called after a pipe to execute a command in the local console, it uses 100% CPU as osm_console poll returns (POLLIN) as if there was something to read from stdin where there isn't. This patch fixes the problem by closing the local console when POLLHUP is received. osm_console will return an error code which will unset the console_init_flag so it won't be reopened. For socket console, cio_close is called to close the current connection but keep the socket open. Signed-off-by: Nicolas Morey-Chaisemartin --- opensm/include/opensm/osm_console.h | 2 +- opensm/include/opensm/osm_console_io.h | 1 + opensm/opensm/main.c | 6 ++++-- opensm/opensm/osm_console.c | 22 ++++++++++++++++++---- opensm/opensm/osm_console_io.c | 2 +- 5 files changed, 25 insertions(+), 8 deletions(-) diff --git a/opensm/include/opensm/osm_console.h b/opensm/include/opensm/osm_console.h index 3ea8fa5..dcce1e4 100644 --- a/opensm/include/opensm/osm_console.h +++ b/opensm/include/opensm/osm_console.h @@ -46,6 +46,6 @@ BEGIN_C_DECLS // TODO replace p_osm -void osm_console(osm_opensm_t * p_osm); +int osm_console(osm_opensm_t * p_osm); END_C_DECLS #endif /* _OSM_CONSOLE_H_ */ diff --git a/opensm/include/opensm/osm_console_io.h b/opensm/include/opensm/osm_console_io.h index 5fe233d..f31d811 100644 --- a/opensm/include/opensm/osm_console_io.h +++ b/opensm/include/opensm/osm_console_io.h @@ -83,6 +83,7 @@ int is_console_enabled(osm_subn_opt_t *p_opt); #ifdef ENABLE_OSM_CONSOLE_SOCKET int cio_open(osm_console_t * p_oct, int new_fd, osm_log_t * p_log); +int cio_close(osm_console_t * p_oct); int is_authorized(osm_console_t * p_oct); #endif diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c index 2271634..5a39f84 100644 --- a/opensm/opensm/main.c +++ b/opensm/opensm/main.c @@ -494,8 +494,10 @@ int osm_manager_loop(osm_subn_opt_t * p_opt, osm_opensm_t * p_osm) Sit here forever - dwell or do console i/o & cmds */ while (!osm_exit_flag) { - if (console_init_flag) - osm_console(p_osm); + if (console_init_flag){ + if(osm_console(p_osm)) + console_init_flag = 0; + } else cl_thread_suspend(10000); diff --git a/opensm/opensm/osm_console.c b/opensm/opensm/osm_console.c index 63c5ea8..04c820c 100644 --- a/opensm/opensm/osm_console.c +++ b/opensm/opensm/osm_console.c @@ -1376,7 +1376,7 @@ static void parse_cmd_line(char *line, osm_opensm_t * p_osm) } } -void osm_console(osm_opensm_t * p_osm) +int osm_console(osm_opensm_t * p_osm) { struct pollfd pollfd[2]; char *p_line; @@ -1409,7 +1409,7 @@ void osm_console(osm_opensm_t * p_osm) } if (poll(fds, nfds, 1000) <= 0) - return; + return 0; #ifdef ENABLE_OSM_CONSOLE_SOCKET if (pollfd[0].revents & POLLIN) { @@ -1422,7 +1422,7 @@ void osm_console(osm_opensm_t * p_osm) "ERR 4B04: Failed to accept console socket: %s\n", strerror(errno)); p_oct->in_fd = -1; - return; + return 0; } if (inet_ntop (AF_INET, &sin.sin_addr, p_oct->client_ip, @@ -1444,7 +1444,7 @@ void osm_console(osm_opensm_t * p_osm) p_oct->client_hn, p_oct->client_ip); close(new_fd); } - return; + return 0; } #endif @@ -1462,5 +1462,19 @@ void osm_console(osm_opensm_t * p_osm) osm_console_exit(p_oct, p_log); if (p_line) free(p_line); + return 0; } + /* input fd is closed (hanged up) */ + if (pollfd[1].revents & POLLHUP) { + /* If we are using a sowket, we close the current connection */ + if (p_oct->socket >= 0) { + cio_close(p_oct); + return 0; + } + /* If we use a local console, stdin is closed (most probable is pipe ended) + * so we close the local console */ + return -1; + } + + return 0; } diff --git a/opensm/opensm/osm_console_io.c b/opensm/opensm/osm_console_io.c index 3d3ece4..ce867bd 100644 --- a/opensm/opensm/osm_console_io.c +++ b/opensm/opensm/osm_console_io.c @@ -94,7 +94,7 @@ int is_console_enabled(osm_subn_opt_t * p_opt) #ifdef ENABLE_OSM_CONSOLE_SOCKET -static int cio_close(osm_console_t * p_oct) +int cio_close(osm_console_t * p_oct) { int rtnval = -1; if (p_oct && (p_oct->in_fd > 0)) { -- 1.6.2-rc2.GIT From nicolas.morey-chaisemartin at ext.bull.net Fri Mar 6 01:28:38 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Fri, 06 Mar 2009 10:28:38 +0100 Subject: [ofa-general] [PATCH] opensm/osm_console_io.h: Modify osm_console_exit so only the connection is killed, not the socket Message-ID: <49B0ECC6.7090700@ext.bull.net> With this patch, calling the exit function from a remote console doesn't kill the socket but only the connection. This way, it will be possible to reconnect top the socket without having to restart opensm. Signed-off-by: Nicolas Morey-Chaisemartin --- opensm/opensm/osm_console_io.c | 12 ++++++++++-- 1 files changed, 10 insertions(+), 2 deletions(-) diff --git a/opensm/opensm/osm_console_io.c b/opensm/opensm/osm_console_io.c index ce867bd..f0cbb8d 100644 --- a/opensm/opensm/osm_console_io.c +++ b/opensm/opensm/osm_console_io.c @@ -210,8 +210,16 @@ int osm_console_init(osm_subn_opt_t * opt, osm_console_t * p_oct, osm_log_t * p_ /* clean up and release resources */ void osm_console_exit(osm_console_t * p_oct, osm_log_t * p_log) { - // clean up and release resources, currently just close the socket - osm_console_close(p_oct, p_log); + + // currently just close the current connection, not the socket +#ifdef ENABLE_OSM_CONSOLE_SOCKET + if ((p_oct->socket > 0) && (p_oct->in_fd != -1)) { + OSM_LOG(p_log, OSM_LOG_INFO, + "Console connection closed: %s (%s)\n", + p_oct->client_hn, p_oct->client_ip); + cio_close(p_oct); + } +#endif } #ifdef ENABLE_OSM_CONSOLE_SOCKET -- 1.6.2-rc2.GIT From sashak at voltaire.com Fri Mar 6 02:12:36 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 12:12:36 +0200 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_switch.h : Fixed wrong comment about return value of osm_switch_set_hops In-Reply-To: <49AE42CD.5070802@ext.bull.net> References: <49AE42CD.5070802@ext.bull.net> Message-ID: <20090306101229.GI6258@sashak.voltaire.com> On 09:58 Wed 04 Mar , Nicolas Morey Chaisemartin wrote: > > Signed-off-by: Nicolas Morey-Chaisemartin Applied. Thanks. Sasha From sashak at voltaire.com Fri Mar 6 02:15:56 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 12:15:56 +0200 Subject: [ofa-general] Re: opensm/libvendor/osm_vendor_ibumad.c: In clear_madw, fix tid endian in message In-Reply-To: <20090304151137.GA3820@comcast.net> References: <20090304151137.GA3820@comcast.net> Message-ID: <20090306101549.GJ6258@sashak.voltaire.com> On 10:11 Wed 04 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Mar 6 02:19:40 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 12:19:40 +0200 Subject: [ofa-general] Re: [PATCH] opensm/osm_req.c: In osm_send_trap144, set producer type according to node type In-Reply-To: <20090304151228.GB3820@comcast.net> References: <20090304151228.GB3820@comcast.net> Message-ID: <20090306101932.GK6258@sashak.voltaire.com> On 10:12 Wed 04 Mar , Hal Rosenstock wrote: > > rather than always set to CA > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Mar 6 02:24:13 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 12:24:13 +0200 Subject: [ofa-general] ***SPAM*** opensm send_trap144 question In-Reply-To: References: Message-ID: <20090306102406.GL6258@sashak.voltaire.com> On 13:50 Wed 04 Mar , Hal Rosenstock wrote: > > clear) the resp_expected flag. It seems to me that a more robust > > strategy would be to set resp_expected to try to make sure that the > > other side sees the trap > > by handling the (trap repress) response matched by transaction ID > > > (and the timeout/retry strategy is used) > > if the trap repress is not received > > > There are a number of underpinnings to change to do this. Would you > > accept a patch along those lines ? It could be done this way if it is a right thing to do. Sasha From sashak at voltaire.com Fri Mar 6 02:33:41 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 12:33:41 +0200 Subject: [ofa-general] Re: [PATCH 3/10] infiniband-diags: convert ibping to "new" ibmad interface In-Reply-To: <20090219190532.faf400f5.weiny2@llnl.gov> References: <20090219190532.faf400f5.weiny2@llnl.gov> Message-ID: <20090306103341.GM6258@sashak.voltaire.com> On 19:05 Thu 19 Feb , Ira Weiny wrote: > --- > infiniband-diags/src/ibping.c | 21 ++++++++---- > libibmad/include/infiniband/mad.h | 66 +++++++++++++++++++++++++------------ > libibmad/src/libibmad.map | 5 +++ > libibmad/src/mad_internal.h | 44 ++++++++++++++++++++++++ When adding new file (src/mad_internal.h) it is also necessary to add it in libibmad_la_SOURCES in Makefile.am (mainly for things like 'make dist', etc). Sasha From sashak at voltaire.com Fri Mar 6 02:56:48 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 12:56:48 +0200 Subject: [ofa-general] Re: [PATCH] opensm/osm_link_mgr.c initialize SMSL In-Reply-To: <49AE496E.1040606@Sun.COM> References: <49AE496E.1040606@Sun.COM> Message-ID: <20090306105648.GN6258@sashak.voltaire.com> Hi Line, On 10:27 Wed 04 Mar , Line.Holen at Sun.COM wrote: > The SMSL is currently not initialized. Whatever is written in PortInfo > at power up is kept. > > The lash routing algorithm makes active use of SL to avoid deadlock in > the fabric. The endnodes needs to be instructed to use the SL calculated > for the endnode-to-SMnode path for SA traffic. If not, the SA traffic > can cause deadlock. > > This change makes sure that the SMSL is properly initialized on all > ports. If flash routing is used, the SMSL is set up based on routing > calculations. For other routing algorithms the default SMSL is used. > > Signed-off-by: Line Holen I cannot even try to apply the patch - all white spaces are mangled there. Please setup your mailer. (For tips you can see MUA sections of http://git.kernel.org/?p=git/git.git;a=blob_plain;f=Documentation/SubmittingPatches) > > --- > > diff --git a/opensm/opensm/osm_link_mgr.c b/opensm/opensm/osm_link_mgr.c > index 37e3e1b..44765e3 100644 > --- a/opensm/opensm/osm_link_mgr.c > +++ b/opensm/opensm/osm_link_mgr.c > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -51,6 +52,49 @@ > #include > #include > #include > +#include > + > +extern uint8_t osm_get_lash_sl(osm_opensm_t * p_osm, > + const osm_port_t * p_src_port, > + const osm_port_t * p_dst_port); This function is used in more than one places, so it can be useful to put its prototype in header file (osm_ucast_lash.h for example). > + > +/********************************************************************** > + **********************************************************************/ > +static uint8_t > +__osm_link_mgr_get_smsl(IN osm_sm_t * sm, > + IN osm_physp_t * const p_physp) '__osm_' prefix is not really needed for static functions. > +{ > + osm_opensm_t *p_osm = sm->p_subn->p_osm; > + const osm_port_t *p_sm_port; > + const osm_port_t *p_src_port; > + ib_net16_t slid; > + ib_net16_t smlid; > + uint8_t sl; > + > + OSM_LOG_ENTER(sm->p_log); > + > + if (p_osm->routing_engine_used != OSM_ROUTING_ENGINE_TYPE_LASH) { > + /* Use default SL if lash routing is not used */ > + OSM_LOG_EXIT(sm->p_log); > + return (OSM_DEFAULT_SL); > + } > + > + /* Find osm_port of the SM itself = dest_port */ > + smlid = sm->p_subn->sm_base_lid; > + p_sm_port = cl_ptr_vector_get(&sm->p_subn->port_lid_tbl, > cl_ntoh16(smlid)); > + > + /* Find osm_port of the source = p_physp */ > + slid = osm_physp_get_base_lid(p_physp); > + p_src_port = cl_ptr_vector_get(&sm->p_subn->port_lid_tbl, > cl_ntoh16(slid)); > + > + /* Call lash to find proper SL */ > + sl = osm_get_lash_sl(p_osm, p_src_port, p_sm_port); > + OSM_LOG(sm->p_log, OSM_LOG_DEBUG, "SL for LID %u to SA is %d\n", > + cl_ntoh16(slid), sl); Please think again do we really need such noisy debug message? The rest looks fine. Although it would be nice to not duplicate code for cases of base port 0 and other ports. Sasha From vlad at lists.openfabrics.org Fri Mar 6 03:18:43 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Fri, 6 Mar 2009 03:18:43 -0800 (PST) Subject: [ofa-general] ofa_1_4_kernel 20090306-0200 daily build status Message-ID: <20090306111843.A40F1E60E78@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From sashak at voltaire.com Fri Mar 6 03:33:50 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 13:33:50 +0200 Subject: [ofa-general] Re: [PATCH] opensm/console: Fixed osm_console poll to handle POLLHUP In-Reply-To: <49B0EA36.9040609@ext.bull.net> References: <49B0EA36.9040609@ext.bull.net> Message-ID: <20090306113350.GO6258@sashak.voltaire.com> On 10:17 Fri 06 Mar , Nicolas Morey Chaisemartin wrote: > When opensm is called after a pipe to execute a command in the local console, > it uses 100% CPU as osm_console poll returns (POLLIN) as if there was something to read from stdin where there isn't. > > This patch fixes the problem by closing the local console when POLLHUP is received. > osm_console will return an error code which will unset the console_init_flag so it won't be reopened. > For socket console, cio_close is called to close the current connection but keep the socket open. > > Signed-off-by: Nicolas Morey-Chaisemartin Applied. Thanks. Sasha From sashak at voltaire.com Fri Mar 6 03:34:09 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 13:34:09 +0200 Subject: [ofa-general] Re: [PATCH] opensm/osm_console_io.h: Modify osm_console_exit so only the connection is killed, not the socket In-Reply-To: <49B0ECC6.7090700@ext.bull.net> References: <49B0ECC6.7090700@ext.bull.net> Message-ID: <20090306113409.GP6258@sashak.voltaire.com> On 10:28 Fri 06 Mar , Nicolas Morey Chaisemartin wrote: > With this patch, calling the exit function from a remote console doesn't kill the socket but only the connection. > This way, it will be possible to reconnect top the socket without having to restart opensm. > > Signed-off-by: Nicolas Morey-Chaisemartin Applied. Thanks. Sasha From sashak at voltaire.com Fri Mar 6 03:41:42 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 13:41:42 +0200 Subject: [ofa-general] Re: [PATCH] fix offset used for parsing of XmtDataSL & RcvDataSL In-Reply-To: References: Message-ID: <20090306114142.GQ6258@sashak.voltaire.com> On 16:45 Thu 05 Mar , Or Gerlitz wrote: > fix the offset used for parsing the fifth SL of the XmtDataSL & RcvDataSL counters > > Signed-off-by: Or Gerlitz Applied. Thanks. Sasha From sashak at voltaire.com Fri Mar 6 03:42:56 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 13:42:56 +0200 Subject: [ofa-general] Re: [PATCH] opensm/infiniband-diags: Changes for C rather than C++ style comments In-Reply-To: <20090227184653.GC15668@comcast.net> References: <20090227184653.GC15668@comcast.net> Message-ID: <20090306114256.GR6258@sashak.voltaire.com> On 13:46 Fri 27 Feb , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From yosefe at Voltaire.COM Fri Mar 6 03:38:53 2009 From: yosefe at Voltaire.COM (Yossi Etigin) Date: Fri, 06 Mar 2009 13:38:53 +0200 Subject: [ofa-general] NFS over RDMA In-Reply-To: <49AFF1C8.2060307@opengridcomputing.com> References: <49AECBE8.4070707@mellanox.com> <49AECE68.2020703@nasa.gov> <49AFF1C8.2060307@opengridcomputing.com> Message-ID: <49B10B4D.5020108@Voltaire.COM> Hi Steve, I've installed OFED-1.4.1-rc1 on rh5.2, and tried to use nfs-utils 1.1.1 to mount with "rdma" option. mount.nfs did not recognize "rdma" option. Also, the "-i" switch which tells to pass mount options to the kernel "as-is" is supported only for kernels 2.6.23 and above (in nfs-utils, utils/mount/mount.c, line 466). So - how to mount nfs-rdma on older kernels, for which there are backports in OFED 1.4.1? Thanks, --Yossi From sashak at voltaire.com Fri Mar 6 03:50:53 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 13:50:53 +0200 Subject: [ofa-general] Re: [PATCHv2] ibsim/sim_net.c: Eliminate IsLEDInfoSupported PortInfo CapabilityMask bit In-Reply-To: <20090303164806.GA9050@comcast.net> References: <20090303164806.GA9050@comcast.net> Message-ID: <20090306115046.GS6258@sashak.voltaire.com> On 11:48 Tue 03 Mar , Hal Rosenstock wrote: > > rather than implement LEDInfo attribute support in the simulator > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Mar 6 03:51:35 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 13:51:35 +0200 Subject: [ofa-general] Re: [PATCH][TRIVIAL] ibsim/sim_mad.c: Eliminate a couple of unneeded blank lines In-Reply-To: <20090303152644.GC4425@comcast.net> References: <20090303152644.GC4425@comcast.net> Message-ID: <20090306115135.GT6258@sashak.voltaire.com> On 10:26 Tue 03 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Mar 6 03:56:06 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 13:56:06 +0200 Subject: [ofa-general] Re: [PATCH] ibsim/sim_mad.c: Better error handling in send_trap In-Reply-To: <20090303191217.GA14839@comcast.net> References: <20090303191217.GA14839@comcast.net> Message-ID: <20090306115559.GU6258@sashak.voltaire.com> On 14:12 Tue 03 Mar , Hal Rosenstock wrote: > > In send_trap, validate trap number prior to determining trap function > Made trapnum parameter to send_trap unsigned > Also, removed unneeded enum init of NO_NODE in NODE_TYPES > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From hnrose at comcast.net Fri Mar 6 04:30:38 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Fri, 6 Mar 2009 07:30:38 -0500 Subject: [ofa-general] ***SPAM*** [PATCHv3] Add pkey table support to osm_get_all_port_attr Message-ID: <20090306123038.GA28792@comcast.net> Only supported in osm_vendor_ibumad.c (separate patch for other vendor layers) Also, update applications using this (osmtest, opensm) Signed-off-by: Hal Rosenstock --- Changes from v2: Added pkey bound check in osm_vendor_get_all_port_attr as pointed out by Sasha Readded initialization in opensm/main.c Changes from v1: Only copy number of pkeys indicated Also, don't indicate insufficient memory error if insufficient pkey space supplied and always return number of pkeys that the port supports Note: initialization prior to get_all_port_attrs call not changed since it is faster this way Other patch for other vendor layers still appropriate following this ibutils patch sent separately diff --git a/opensm/libvendor/osm_vendor_ibumad.c b/opensm/libvendor/osm_vendor_ibumad.c index d7089f0..89b3c43 100644 --- a/opensm/libvendor/osm_vendor_ibumad.c +++ b/opensm/libvendor/osm_vendor_ibumad.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -556,12 +557,13 @@ osm_vendor_get_all_port_attr(IN osm_vendor_t * const p_vend, umad_ca_t ca; ib_port_attr_t *attr = p_attr_array; unsigned done = 0; - int r, i, j; + int r, i, j, k; OSM_LOG_ENTER(p_vend->p_log); CL_ASSERT(p_vend && p_num_ports); + r = 0; if (!*p_num_ports) { r = IB_INVALID_PARAMETER; OSM_LOG(p_vend->p_log, OSM_LOG_ERROR, "ERR 5418: " @@ -576,9 +578,7 @@ osm_vendor_get_all_port_attr(IN osm_vendor_t * const p_vend, } for (i = 0; i < p_vend->ca_count && !done; i++) { - /* - * For each CA, retrieve the port guids - */ + /* For each CA, retrieve the port attributes */ if (umad_get_ca(p_vend->ca_names[i], &ca) == 0) { if (ca.node_type < 1 || ca.node_type > 3) continue; @@ -590,6 +590,14 @@ osm_vendor_get_all_port_attr(IN osm_vendor_t * const p_vend, attr->port_num = ca.ports[j]->portnum; attr->sm_lid = ca.ports[j]->sm_lid; attr->link_state = ca.ports[j]->state; + if (attr->num_pkeys && attr->p_pkey_table) { + if (attr->num_pkeys > ca.ports[j]->pkeys_size) + attr->num_pkeys = ca.ports[j]->pkeys_size; + for (k = 0; k < attr->num_pkeys; k++) + attr->p_pkey_table[k] = + cl_hton16(ca.ports[j]->pkeys[k]); + } + attr->num_pkeys = ca.ports[j]->pkeys_size; attr++; if (attr - p_attr_array > *p_num_ports) { done = 1; @@ -601,7 +609,6 @@ osm_vendor_get_all_port_attr(IN osm_vendor_t * const p_vend, } *p_num_ports = attr - p_attr_array; - r = 0; Exit: OSM_LOG_EXIT(p_vend->p_log); diff --git a/opensm/opensm/main.c b/opensm/opensm/main.c index 4d80120..296d5d5 100644 --- a/opensm/opensm/main.c +++ b/opensm/opensm/main.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -64,7 +65,7 @@ volatile unsigned int osm_exit_flag = 0; static volatile unsigned int osm_hup_flag = 0; static volatile unsigned int osm_usr1_flag = 0; -#define GUID_ARRAY_SIZE 64 +#define MAX_LOCAL_IBPORTS 64 #define INVALID_GUID (0xFFFFFFFFFFFFFFFFULL) static void mark_exit_flag(int signum) @@ -369,15 +370,17 @@ static void show_usage(void) **********************************************************************/ static ib_net64_t get_port_guid(IN osm_opensm_t * p_osm, uint64_t port_guid) { - ib_port_attr_t attr_array[GUID_ARRAY_SIZE]; - uint32_t num_ports = GUID_ARRAY_SIZE; + ib_port_attr_t attr_array[MAX_LOCAL_IBPORTS]; + uint32_t num_ports = MAX_LOCAL_IBPORTS; uint32_t i, choice = 0; ib_api_status_t status; - /* - Call the transport layer for a list of local port - GUID values. - */ + for (i = 0; i < num_ports; i++) { + attr_array[i].num_pkeys = 0; + attr_array[i].p_pkey_table = NULL; + } + + /* Call the transport layer for a list of local port GUID values */ status = osm_vendor_get_all_port_attr(p_osm->p_vendor, attr_array, &num_ports); if (status != IB_SUCCESS) { diff --git a/opensm/osmtest/main.c b/opensm/osmtest/main.c index 4ca99f3..83c1e13 100644 --- a/opensm/osmtest/main.c +++ b/opensm/osmtest/main.c @@ -1,6 +1,7 @@ /* * Copyright (c) 2004-2006 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -54,7 +55,7 @@ Future versions could support multiple subents by instantiating more than one osmtest object. */ -#define GUID_ARRAY_SIZE 64 +#define MAX_LOCAL_IBPORTS 64 #define OSMT_DEFAULT_RETRY_COUNT 3 #define OSMT_DEFAULT_TRANS_TIMEOUT_MILLISEC 1000 #define OSMT_DEFAULT_TRAP_WAIT_TIMEOUT_SEC 10 @@ -73,6 +74,8 @@ boolean_t osmt_is_debug(void) /********************************************************************** **********************************************************************/ +void show_usage(void); + void show_usage() { printf @@ -207,13 +210,19 @@ void show_usage() /********************************************************************** **********************************************************************/ +static void print_all_guids(IN osmtest_t * p_osmt); static void print_all_guids(IN osmtest_t * p_osmt) { ib_api_status_t status; - uint32_t num_ports = GUID_ARRAY_SIZE; - ib_port_attr_t attr_array[GUID_ARRAY_SIZE]; + uint32_t num_ports = MAX_LOCAL_IBPORTS; + ib_port_attr_t attr_array[MAX_LOCAL_IBPORTS]; int i; + for (i = 0; i < num_ports; i++) { + attr_array[i].num_pkeys = 0; + attr_array[i].p_pkey_table = NULL; + } + /* Call the transport layer for a list of local port GUID values. @@ -238,10 +247,15 @@ static void print_all_guids(IN osmtest_t * p_osmt) ib_net64_t get_port_guid(IN osmtest_t * p_osmt, uint64_t port_guid) { ib_api_status_t status; - uint32_t num_ports = GUID_ARRAY_SIZE; - ib_port_attr_t attr_array[GUID_ARRAY_SIZE]; + uint32_t num_ports = MAX_LOCAL_IBPORTS; + ib_port_attr_t attr_array[MAX_LOCAL_IBPORTS]; int i; + for (i = 0; i < num_ports; i++) { + attr_array[i].num_pkeys = 0; + attr_array[i].p_pkey_table = NULL; + } + /* Call the transport layer for a list of local port GUID values. diff --git a/opensm/osmtest/osmtest.c b/opensm/osmtest/osmtest.c index 32cfa01..986a8d2 100644 --- a/opensm/osmtest/osmtest.c +++ b/opensm/osmtest/osmtest.c @@ -2,6 +2,7 @@ * Copyright (c) 2006-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -63,7 +64,7 @@ #endif #define POOL_MIN_ITEMS 64 -#define GUID_ARRAY_SIZE 64 +#define MAX_LOCAL_IBPORTS 64 typedef struct _osmtest_sm_info_rec { ib_net64_t sm_guid; @@ -7094,11 +7095,17 @@ osmtest_bind(IN osmtest_t * p_osmt, { uint32_t port_index; ib_api_status_t status; - uint32_t num_ports = GUID_ARRAY_SIZE; - ib_port_attr_t attr_array[GUID_ARRAY_SIZE]; + uint32_t num_ports = MAX_LOCAL_IBPORTS; + ib_port_attr_t attr_array[MAX_LOCAL_IBPORTS]; + int i; OSM_LOG_ENTER(&p_osmt->log); + for (i = 0; i < num_ports; i++) { + attr_array[i].num_pkeys = 0; + attr_array[i].p_pkey_table = NULL; + } + /* * Call the transport layer for a list of local port * GUID values. From sashak at voltaire.com Fri Mar 6 04:45:47 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 14:45:47 +0200 Subject: [ofa-general] Re: [PATCH] [ib-diag] ibstat: port_dump is called too many times In-Reply-To: <40F59C0D0E0E47F99E3D21458877BF31@amr.corp.intel.com> References: <40F59C0D0E0E47F99E3D21458877BF31@amr.corp.intel.com> Message-ID: <20090306124547.GV6258@sashak.voltaire.com> Hi Sean, On 13:15 Tue 03 Mar , Sean Hefty wrote: > portnum index should go from 0 to numports - 1. At least in linux it is not so. libibumad initializes numports as max port available for this card (normally on hosts ports 1 and 2 are initialized). > This fixes an > issue on WinOF where a bogus last port is displayed. > > Signed-off-by: Sean Hefty > --- > > infiniband-diags/src/ibstat.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > diff --git a/infiniband-diags/src/ibstat.c b/infiniband-diags/src/ibstat.c > index 7985be1..41faea8 100644 > --- a/infiniband-diags/src/ibstat.c > +++ b/infiniband-diags/src/ibstat.c > @@ -146,7 +146,7 @@ ca_stat(char *ca_name, int portnum, int no_ports) > if (no_ports) > return 0; > > - for (portnum = 0; portnum <= ca.numports; portnum++) > + for (portnum = 0; portnum < ca.numports; portnum++) > port_dump(ca.ports[portnum], 0); As expected on linux it shows only first port of two ports CA. Likely we have libibumad implemented differently in this part. Sasha From Line.Holen at Sun.COM Fri Mar 6 04:53:06 2009 From: Line.Holen at Sun.COM (Line.Holen at Sun.COM) Date: Fri, 06 Mar 2009 13:53:06 +0100 Subject: [ofa-general] Re: [PATCH] opensm/osm_link_mgr.c initialize SMSL In-Reply-To: <20090306105648.GN6258@sashak.voltaire.com> References: <49AE496E.1040606@Sun.COM> <20090306105648.GN6258@sashak.voltaire.com> Message-ID: <49B11CB2.6060002@Sun.COM> On 03/ 6/09 11:56 AM, Sasha Khapyorsky wrote: > Hi Line, > > On 10:27 Wed 04 Mar , Line.Holen at Sun.COM wrote: > >> The SMSL is currently not initialized. Whatever is written in PortInfo >> at power up is kept. >> >> The lash routing algorithm makes active use of SL to avoid deadlock in >> the fabric. The endnodes needs to be instructed to use the SL calculated >> for the endnode-to-SMnode path for SA traffic. If not, the SA traffic >> can cause deadlock. >> >> This change makes sure that the SMSL is properly initialized on all >> ports. If flash routing is used, the SMSL is set up based on routing >> calculations. For other routing algorithms the default SMSL is used. >> >> Signed-off-by: Line Holen >> > > I cannot even try to apply the patch - all white spaces are mangled > there. Please setup your mailer. (For tips you can see MUA sections of > http://git.kernel.org/?p=git/git.git;a=blob_plain;f=Documentation/SubmittingPatches) > OK, I'll look into this. BTW, is there any policy with respect to use of tab for indentations ? > >> --- >> >> diff --git a/opensm/opensm/osm_link_mgr.c b/opensm/opensm/osm_link_mgr.c >> index 37e3e1b..44765e3 100644 >> --- a/opensm/opensm/osm_link_mgr.c >> +++ b/opensm/opensm/osm_link_mgr.c >> @@ -2,6 +2,7 @@ >> * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. >> * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. >> * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. >> + * Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved. >> * >> * This software is available to you under a choice of one of two >> * licenses. You may choose to be licensed under the terms of the GNU >> @@ -51,6 +52,49 @@ >> #include >> #include >> #include >> +#include >> + >> +extern uint8_t osm_get_lash_sl(osm_opensm_t * p_osm, >> + const osm_port_t * p_src_port, >> + const osm_port_t * p_dst_port); >> > > This function is used in more than one places, so it can be useful to > put its prototype in header file (osm_ucast_lash.h for example). > OK. > >> + >> +/********************************************************************** >> + **********************************************************************/ >> +static uint8_t >> +__osm_link_mgr_get_smsl(IN osm_sm_t * sm, >> + IN osm_physp_t * const p_physp) >> > > '__osm_' prefix is not really needed for static functions. > This prefix is used for all the other static functions in this file, but I can remove it if you like. > >> +{ >> + osm_opensm_t *p_osm = sm->p_subn->p_osm; >> + const osm_port_t *p_sm_port; >> + const osm_port_t *p_src_port; >> + ib_net16_t slid; >> + ib_net16_t smlid; >> + uint8_t sl; >> + >> + OSM_LOG_ENTER(sm->p_log); >> + >> + if (p_osm->routing_engine_used != OSM_ROUTING_ENGINE_TYPE_LASH) { >> + /* Use default SL if lash routing is not used */ >> + OSM_LOG_EXIT(sm->p_log); >> + return (OSM_DEFAULT_SL); >> + } >> + >> + /* Find osm_port of the SM itself = dest_port */ >> + smlid = sm->p_subn->sm_base_lid; >> + p_sm_port = cl_ptr_vector_get(&sm->p_subn->port_lid_tbl, >> cl_ntoh16(smlid)); >> + >> + /* Find osm_port of the source = p_physp */ >> + slid = osm_physp_get_base_lid(p_physp); >> + p_src_port = cl_ptr_vector_get(&sm->p_subn->port_lid_tbl, >> cl_ntoh16(slid)); >> + >> + /* Call lash to find proper SL */ >> + sl = osm_get_lash_sl(p_osm, p_src_port, p_sm_port); >> + OSM_LOG(sm->p_log, OSM_LOG_DEBUG, "SL for LID %u to SA is %d\n", >> + cl_ntoh16(slid), sl); >> > > Please think again do we really need such noisy debug message? > It can be removed. > The rest looks fine. Although it would be nice to not duplicate code for > cases of base port 0 and other ports. > > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Fri Mar 6 05:17:01 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 15:17:01 +0200 Subject: [ofa-general] Re: complib and cl_nodenamemap In-Reply-To: <49E9FBB3AEB64CEDA3577778C69E6392@amr.corp.intel.com> References: <49E9FBB3AEB64CEDA3577778C69E6392@amr.corp.intel.com> Message-ID: <20090306131701.GW6258@sashak.voltaire.com> Hi Sean, On 11:13 Tue 03 Mar , Sean Hefty wrote: > > I'd like to see about doing 'something' with nodenamemap. Currently, it's part > of complib, but the interfaces and operation are not similar to other complib > type abstractions. Yes, it was added later. And main reason for placing it into complib was to share this code between OpenSM and infiniband-diags. > From what I can tell, nodenamemap performs two abstractions. At a lower level, > it implements functionality similar to cl_map. The main difference is that > cl_map requires the user to allocate space for the object being inserted into > the map. If we add a new call to complib, we can provide an abstraction of the > functionality that nodenamemap requires: > > cl_map_insert_copy(cl_map_t *p_map, uint64_t key, > void *p_object, size_t object_size); > > There would be internal changes needed to free the object, plus changes to > cl_map_remove return values for copied objects. I don't know how it will be generally useful. Basically we are trying to not use cl_map in OpenSM but instead much faster cl_qmap primitives. > At a higher level, nodenamemap parses an application specific file and stores > the contents of the file into a map. I'm not sure that this functionality > really belongs as a part of complib. But regardless, it makes more sense to me > to separate the parsing of the file from maintaining the data in a map. > > There's not a need to abstract opening or closing the file. So the only > functionality that's needed is parsing a line of input from the file. Each > application could link in the necessary code directly, the parsing code could go > into a new 'ibcommon' library (not really worth it for 1 call), New library? I'm not really happy. If we need "generic" library what is wrong with using complib then? > or the file > format just needs to be generic enough to work with a wide variety of > applications (and maybe it already is). It used mainly in OpenSM for various files parsing with similar format. Also in infiniband-diags. > Is it acceptable to change or remove the nodenamemap abstraction from complib, > and if so, is there a specific direction to take? Basically yes. But we need to find some acceptable solution first which will be better than how it is implemented now. Sasha From sashak at voltaire.com Fri Mar 6 05:21:08 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 15:21:08 +0200 Subject: [ofa-general] Re: [PATCH] opensm/osm_link_mgr.c initialize SMSL In-Reply-To: <49B11CB2.6060002@Sun.COM> References: <49AE496E.1040606@Sun.COM> <20090306105648.GN6258@sashak.voltaire.com> <49B11CB2.6060002@Sun.COM> Message-ID: <20090306132108.GX6258@sashak.voltaire.com> On 13:53 Fri 06 Mar , Line.Holen at Sun.COM wrote: >> >> I cannot even try to apply the patch - all white spaces are mangled >> there. Please setup your mailer. (For tips you can see MUA sections of >> http://git.kernel.org/?p=git/git.git;a=blob_plain;f=Documentation/SubmittingPatches) >> > OK, I'll look into this. BTW, is there any policy with respect to use of > tab for indentations ? Yes, we are using tab character for indentation. Look at opensm/doc/opensm-coding-style.txt (and eventually at your /usr/src/linux/Documentation/CodingStyle). Sasha From sashak at voltaire.com Fri Mar 6 05:51:42 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 15:51:42 +0200 Subject: [ofa-general] Re: [PATCH] ibsim/sim_mad.c: Add trap/trap repress handling in simulated SMA In-Reply-To: <20090305122318.GA20341@comcast.net> References: <20090305122318.GA20341@comcast.net> Message-ID: <20090306135142.GY6258@sashak.voltaire.com> On 07:23 Thu 05 Mar , Hal Rosenstock wrote: > > Add notice attribute support > Also, changes to decode_sim_MAD and reply_MAD for trap/trap repress methods > > Signed-off-by: Hal Rosenstock > > --- > diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c > index 32a667e..1a929c5 100644 > --- a/ibsim/sim_mad.c > +++ b/ibsim/sim_mad.c > @@ -57,7 +57,7 @@ > typedef int (Smpfn) (Port * port, unsigned op, uint32_t mod, uint8_t * data); > typedef int (EncodeTrapfn) (Port * port, char *data); > > -static Smpfn do_nodeinfo, do_nodedesc, do_switchinfo, do_portinfo, > +static Smpfn do_notice, do_nodeinfo, do_nodedesc, do_switchinfo, do_portinfo, > do_linearforwtbl, do_multicastforwtbl, do_portcounters, do_extcounters, > do_pkeytbl, do_sl2vl, do_vlarb, do_guidinfo, do_cpi; > > @@ -65,7 +65,8 @@ static EncodeTrapfn encode_trap128; > static EncodeTrapfn encode_trap144; > > static Smpfn *attrs[IB_PERFORMANCE_CLASS + 1][0xff] = { > - [IB_SMI_CLASS] {[IB_ATTR_NODE_DESC] do_nodedesc, > + [IB_SMI_CLASS] {[NOTICE] do_notice, Now Trap MAD is routed to SM, Right? By having do_notice() processor ibsim will reply by itself and will prevent SM from receiving traps. No? > + [IB_ATTR_NODE_DESC] do_nodedesc, > [IB_ATTR_NODE_INFO] do_nodeinfo, > [IB_ATTR_SWITCH_INFO] do_switchinfo, > [IB_ATTR_PORT_INFO] do_portinfo, > @@ -118,15 +119,22 @@ static int decode_sim_MAD(Client * cl, struct sim_request * r, ib_rpc_t * rpc, > { > void *buf = r->mad; > int response; > + uint8_t method; > > - // first word > - response = mad_get_field(buf, 0, IB_MAD_RESPONSE_F); > + method = mad_get_field(buf, 0, IB_MAD_METHOD_F); > + if (method == IB_MAD_METHOD_TRAP_REPRESS) > + response = 1; > + else if (method == IB_MAD_METHOD_TRAP) > + response = 0; > + else > + // first word > + response = mad_get_field(buf, 0, IB_MAD_RESPONSE_F); TrapRepress MAD is dropped by ibsim (in process_packet() function right after decode_sim_MAD() call). How will "response" return value matter there? (With trap method the original code should work fine - bit 7 of method field is '0'). Sasha > if (mad_get_field(buf, 0, IB_MAD_CLASSVER_F) > 2 || // sma ver is 1, sa is 2 > mad_get_field(buf, 0, IB_MAD_BASEVER_F) != 1) { > IBWARN("bad smp headers (1st word)"); > return -1; > } > - rpc->method = mad_get_field(buf, 0, IB_MAD_METHOD_F); > + rpc->method = method; > rpc->mgtclass = mad_get_field(buf, 0, IB_MAD_MGMTCLASS_F); > > // second word: > @@ -189,11 +197,17 @@ static int forward_MAD(void *buf, ib_rpc_t * rpc, ib_dr_path_t * path) > static int reply_MAD(void *buf, ib_rpc_t * rpc, ib_dr_path_t * path, > int status, void *data) > { > - // first word > - mad_set_field(buf, 0, IB_MAD_RESPONSE_F, 1); > - mad_set_field(buf, 0, IB_MAD_METHOD_F, 0x81); // SUBN_GETRESP > - > - // second word: > + uint8_t method; > + > + method = mad_get_field(buf, 0, IB_MAD_METHOD_F); > + if (method == IB_MAD_METHOD_TRAP) > + mad_set_field(buf, 0, IB_MAD_METHOD_F, IB_MAD_METHOD_TRAP_REPRESS); > + else if (method != IB_MAD_METHOD_TRAP_REPRESS) { > + // first word > + mad_set_field(buf, 0, IB_MAD_RESPONSE_F, 1); > + mad_set_field(buf, 0, IB_MAD_METHOD_F, 0x81); // SUBN_GETRESP > + } > + // second word > if (rpc->mgtclass == 0x81) { // direct route > mad_set_field(buf, 0, IB_DRSMP_STATUS_F, status); > mad_set_field(buf, 0, IB_DRSMP_DIRECTION_F, 1); > @@ -218,6 +232,16 @@ static int reply_MAD(void *buf, ib_rpc_t * rpc, ib_dr_path_t * path, > return 0; > } > > +static int do_notice(Port * port, unsigned op, uint32_t mod, uint8_t * data) > +{ > + int status = 0; > + > + if (op != IB_MAD_METHOD_TRAP && op != IB_MAD_METHOD_TRAP_REPRESS) > + status = ERR_METHOD_UNSUPPORTED; > + > + return status; > +} > + > static int do_cpi(Port * port, unsigned op, uint32_t mod, uint8_t * data) > { > int status = 0; From PHF at zurich.ibm.com Fri Mar 6 07:03:50 2009 From: PHF at zurich.ibm.com (Philip Frey1) Date: Fri, 6 Mar 2009 16:03:50 +0100 Subject: [ofa-general] NFS over RDMA In-Reply-To: <49AFF1C8.2060307@opengridcomputing.com> References: <49AECBE8.4070707@mellanox.com> <49AECE68.2020703@nasa.gov> <49AFF1C8.2060307@opengridcomputing.com> Message-ID: Steve, all, many thanks for your help! I am running a Fedora provided kernel (see below) and OFED-1.4. Running 'install.pl' in 'custom' mode did never ask about NFS. Neither did setting the 'nfsrdma=y' change anything. 'find' told me that there is nothing in OFED-1.4 that contains 'nfs' but there are nfs files in the release candidates. I will first try OFED-1.4.1 on my Fedora 9 (2.6.27.12-78.2.8.fc9.x86_64) and move to the 2.6.28 kernel provided by kernel.org if it does not work. Cheers, Philip From: Steve Wise To: Jeff Becker Cc: Philip Frey1 , Tobias Abt1/Zurich/IBM at IBMCH, "general at lists.openfabrics.org" Date: 03/05/2009 04:36 PM Subject: Re: [ofa-general] NFS over RDMA The stuff shipped in ofed-1.4 isn't very stable. I suggest either ofed-1.4.1-rc1 or just a stock 2.6.28 kernel (see my earlier email). Steve. Jeff Becker wrote: > Hi. If you use install.pl, I think you need to select custom install > (not "install all"). This will allow you to pick the nfsrdma option. > > -jeff > > Vu Pham wrote: > >> The easiest steps to have nfsrdma up and running: >> Download OFED-1.4.1-rc1 from >> http://www.openfabrics.org/downloads/OFED/ofed-1.4.1/ >> and build/install nfsrdma. It is running on 2.6.27/26/25 and RHEL 5.2 >> default kernels >> >> Or >> + Download kernel 2.6.27 from kernel.org >> + Build the kernel 2.6.27 with IB driver (and other modules ie. >> user_mad, addr_trans, ib_mthca, mlx4_ib... ), nfs client/server, >> sunrpc... as modules >> + Boot up with new kernel >> + Follow the attached README >> >> NOTE: You can also download and run nfsrdma on latest development kernel >> 2.6.29-rc7; however, it does not work on Mellanox ConnectX HCA due to a >> bug on Fast memory registration work request. We are debugging it now >> >> -vu >> >> >>> Hello, >>> >>> I want to try NFS over RDMA. According to the documentation enclosed >>> within the OFED-1.4 release, it should be as simple as setting >>> "nfsrdma=y". >>> >>> Unfortunately there is no such option. The interactive menu offered by >>> './install.pl' does not mention NFS neither. >>> >>> Is there an updated version of the documentation on how to run NFS over >>> RDMA? I have noticed, that it is part of the vanilla Linux kernel but >>> I still did >>> not manage to get it up and running. >>> >>> Many thanks for your advice, >>> Philip >>> >>> -- >>> Philip Frey >>> IBM Zurich Research Laboratory >>> Saumerstrasse 4 | Phone: +41 44 >>> 724 8613 >>> CH-8803 Rueschlikon/Switzerland | Email: phf at zurich.ibm.com >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> general mailing list >>> general at lists.openfabrics.org >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>> >>> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general >>> >>> >> >> > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > _______________________________________________ general mailing list general at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -------------- next part -------------- An HTML attachment was scrubbed... URL: From hnrose at comcast.net Fri Mar 6 07:25:54 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Fri, 6 Mar 2009 10:25:54 -0500 Subject: [ofa-general] [PATCH] opensm/PerfMgr: A few more esp0 changes Message-ID: <20090306152554.GA29312@comcast.net> Also, some cosmetic changes Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_perfmgr.c b/opensm/opensm/osm_perfmgr.c index c5c4572..4a6f65c 100644 --- a/opensm/opensm/osm_perfmgr.c +++ b/opensm/opensm/osm_perfmgr.c @@ -469,14 +469,13 @@ static void __osm_perfmgr_query_counters(cl_map_item_t * const p_map_item, void *context) { ib_api_status_t status = IB_SUCCESS; - uint8_t port = 0, startport = 1; osm_perfmgr_t *pm = (osm_perfmgr_t *) context; osm_node_t *node = NULL; __monitored_node_t *mon_node = (__monitored_node_t *) p_map_item; osm_madw_context_t mad_context; - uint8_t num_ports = 0; uint64_t node_guid = 0; ib_net32_t remote_qp; + uint8_t port, num_ports = 0; OSM_LOG_ENTER(pm->log); @@ -505,12 +504,8 @@ __osm_perfmgr_query_counters(cl_map_item_t * const p_map_item, void *context) goto Exit; } - /* check for switch enhanced port 0 */ - if (mon_node->esp0) - startport = 0; - /* issue the query for each port */ - for (port = startport; port < num_ports; port++) { + for (port = (mon_node->esp0) ? 0 : 1; port < num_ports; port++) { ib_net16_t lid; if (!osm_node_get_physp_ptr(node, port)) diff --git a/opensm/opensm/osm_perfmgr_db.c b/opensm/opensm/osm_perfmgr_db.c index 78580a6..b0b2e4a 100644 --- a/opensm/opensm/osm_perfmgr_db.c +++ b/opensm/opensm/osm_perfmgr_db.c @@ -367,7 +367,7 @@ debug_dump_dc_reading(perfmgr_db_t * db, uint64_t guid, uint8_t port_num, { osm_log_t *log = db->perfmgr->log; if (!osm_log_is_active(log, OSM_LOG_DEBUG)) - return; /* optimize this a big */ + return; osm_log(log, OSM_LOG_DEBUG, "xd %" PRIu64 " <-- %" PRIu64 " (%" PRIu64 ")\n", @@ -565,8 +565,9 @@ static void __dump_node_mr(_db_node_t * node, FILE * fp) "rcv_pkts", "unicast_xmit_pkts", "unicast_rcv_pkts", - "multicast_xmit_pkts", "multicast_rcv_pkts"); - for (i = 1; i < node->num_ports; i++) { + "multicast_xmit_pkts", + "multicast_rcv_pkts"); + for (i = (node->esp0) ? 0 : 1; i < node->num_ports; i++) { char *since = ctime(&node->ports[i].last_reset); since[strlen(since) - 1] = '\0'; /* remove \n */ @@ -609,7 +610,7 @@ static void __dump_node_hr(_db_node_t * node, FILE * fp) int i = 0; fprintf(fp, "\n"); - for (i = 1; i < node->num_ports; i++) { + for (i = (node->esp0) ? 0 : 1; i < node->num_ports; i++) { char *since = ctime(&node->ports[i].last_reset); since[strlen(since) - 1] = '\0'; /* remove \n */ @@ -719,7 +720,7 @@ done: void perfmgr_db_print_by_guid(perfmgr_db_t * db, uint64_t nodeguid, FILE *fp) { - cl_map_item_t *node = NULL; + cl_map_item_t *node; cl_plock_acquire(&db->lock); @@ -727,7 +728,7 @@ perfmgr_db_print_by_guid(perfmgr_db_t * db, uint64_t nodeguid, FILE *fp) if (node != cl_qmap_end(&db->pc_data)) __dump_node_hr((_db_node_t *)node, fp); else - fprintf(fp, "Node %"PRIx64" not found...\n", nodeguid); + fprintf(fp, "Node 0x%" PRIx64 " not found...\n", nodeguid); cl_plock_release(&db->lock); } From jon at opengridcomputing.com Fri Mar 6 07:39:24 2009 From: jon at opengridcomputing.com (Jon Mason) Date: Fri, 6 Mar 2009 09:39:24 -0600 Subject: [ofa-general] NFS over RDMA In-Reply-To: <49B10B4D.5020108@Voltaire.COM> References: <49AECBE8.4070707@mellanox.com> <49AECE68.2020703@nasa.gov> <49AFF1C8.2060307@opengridcomputing.com> <49B10B4D.5020108@Voltaire.COM> Message-ID: <20090306153920.GA18292@opengridcomputing.com> On Fri, Mar 06, 2009 at 01:38:53PM +0200, Yossi Etigin wrote: > Hi Steve, > > I've installed OFED-1.4.1-rc1 on rh5.2, and tried to use nfs-utils 1.1.1 > to mount with "rdma" option. mount.nfs did not recognize "rdma" option. > Also, the "-i" switch > which tells to pass mount options to the kernel "as-is" > is supported only for kernels 2.6.23 and above (in nfs-utils, > utils/mount/mount.c, line 466). > > So - how to mount nfs-rdma on older kernels, for which there > are backports in OFED 1.4.1? There is a version of nfs-utils that I modified to allow older kernels to mount using rdma. I am in the process of turning it into a SRPM, and should have something out later today for inclusion into OFED. Thanks, Jon > > Thanks, > --Yossi > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From sashak at voltaire.com Fri Mar 6 09:30:26 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 19:30:26 +0200 Subject: [ofa-general] Re: [PATCH 1/2] opensm: Handle trap repress on trap 144 generation In-Reply-To: <20090305122433.GB20341@comcast.net> References: <20090305122433.GB20341@comcast.net> Message-ID: <20090306173026.GA9673@sashak.voltaire.com> Hi Hal, On 07:24 Thu 05 Mar , Hal Rosenstock wrote: > > Make trap generation sending be a transaction (expect a response) > and handle incoming trap repress (notice attribute) I don't see how this works. See below. > ib_mad_is_response is now changed to better reflect responses > (not just R bit in MAD header) > > Signed-off-by: Hal Rosenstock > @@ -3797,8 +3810,10 @@ static inline boolean_t OSM_API > ib_mad_is_response(IN const ib_mad_t * const p_mad) > { > CL_ASSERT(p_mad); > - return ((p_mad->method & IB_MAD_METHOD_RESP_MASK) == > - IB_MAD_METHOD_RESP_MASK); > + return (p_mad->method & IB_MAD_METHOD_RESP_MASK || > + p_mad->method == IB_MAD_METHOD_TRAP_REPRESS || > + (p_mad->mgmt_class == IB_MCLASS_BM && > + p_mad->attr_mod & IB_BM_ATTR_MOD_RESP)); Why to mix things? BR is not used anywhere in OpenSM. > } > > /* > diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c > index 2e851b2..1a46e50 100644 > --- a/opensm/opensm/osm_req.c > +++ b/opensm/opensm/osm_req.c > @@ -246,6 +246,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) > > madw->mad_addr.dest_lid = pi->master_sm_base_lid; > madw->mad_addr.addr_type.smi.source_lid = pi->base_lid; > + madw->resp_expected = TRUE; > madw->fail_msg = CL_DISP_MSGID_NONE; > > smp = osm_madw_get_smp_ptr(madw); > diff --git a/opensm/opensm/osm_sm_mad_ctrl.c b/opensm/opensm/osm_sm_mad_ctrl.c > index 267ec85..a9124fa 100644 > --- a/opensm/opensm/osm_sm_mad_ctrl.c > +++ b/opensm/opensm/osm_sm_mad_ctrl.c > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2009 HNR Consulting. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -250,10 +251,12 @@ __osm_sm_mad_ctrl_process_get_resp(IN osm_sm_mad_ctrl_t * const p_ctrl, > case IB_MAD_ATTR_P_KEY_TABLE: > msg_id = OSM_MSG_MAD_PKEY; > break; > + case IB_MAD_ATTR_NOTICE: > + msg_id = OSM_MSG_MAD_NOTICE; > + break; This is GetResp processing, not TrapRepress. And here osm_trap_rcv_process() is actually scheduled for execution. > > case IB_MAD_ATTR_GUID_INFO: > case IB_MAD_ATTR_CLASS_PORT_INFO: > - case IB_MAD_ATTR_NOTICE: > case IB_MAD_ATTR_INFORM_INFO: > default: > cl_atomic_inc(&p_ctrl->p_stats->qp0_mads_rcvd_unknown); > @@ -537,6 +540,57 @@ Exit: > * SEE ALSO > *********/ > > +/****f* opensm: SM/__osm_sm_mad_ctrl_process_trap_repress > + * NAME > + * __osm_sm_mad_ctrl_process_trap_repress > + * > + * DESCRIPTION > + * This function handles method TrapRepress() for received MADs. > + * > + * SYNOPSIS > + */ > +static void > +__osm_sm_mad_ctrl_process_trap_repress(IN osm_sm_mad_ctrl_t * const p_ctrl, > + IN osm_madw_t * p_madw) > +{ > + ib_smp_t *p_smp; > + > + OSM_LOG_ENTER(p_ctrl->p_log); > + > + p_smp = osm_madw_get_smp_ptr(p_madw); > + > + /* > + Note that attr_id (like the rest of the MAD) is in > + network byte order. > + */ > + switch (p_smp->attr_id) { > + case IB_MAD_ATTR_NOTICE: > + break; > + > + default: > + cl_atomic_inc(&p_ctrl->p_stats->qp0_mads_rcvd_unknown); > + OSM_LOG(p_ctrl->p_log, OSM_LOG_ERROR, "ERR 3105: " > + "Unsupported attribute = 0x%X\n", > + cl_ntoh16(p_smp->attr_id)); > + osm_dump_dr_smp(p_ctrl->p_log, p_smp, OSM_LOG_ERROR); > + break; > + } > + > + osm_mad_pool_put(p_ctrl->p_mad_pool, p_madw); > + > + OSM_LOG_EXIT(p_ctrl->p_log); > +} And this function which should process TrapRepress method does nothing, right? > + > +/* > + * PARAMETERS > + * > + * RETURN VALUES > + * > + * NOTES > + * > + * SEE ALSO > + *********/ > + > /****f* opensm: SM/__osm_sm_mad_ctrl_rcv_callback > * NAME > * __osm_sm_mad_ctrl_rcv_callback > @@ -624,10 +678,14 @@ __osm_sm_mad_ctrl_rcv_callback(IN osm_madw_t * p_madw, This (__osm_sm_mad_ctrl_rcv_callback()) function has above if (ib_smp_is_response(p_smp) || (p_smp->method == IB_MAD_METHOD_TRAP_REPRESS)) check. Following your ib_mad_is_response() modification it should be simplified (probably in other places too). Sasha > __osm_sm_mad_ctrl_process_set(p_ctrl, p_madw); > break; > > + case IB_MAD_METHOD_TRAP_REPRESS: > + CL_ASSERT(p_req_madw == NULL); > + __osm_sm_mad_ctrl_process_trap_repress(p_ctrl, p_madw); > + break; > + > case IB_MAD_METHOD_SEND: > case IB_MAD_METHOD_REPORT: > case IB_MAD_METHOD_REPORT_RESP: > - case IB_MAD_METHOD_TRAP_REPRESS: > default: > cl_atomic_inc(&p_ctrl->p_stats->qp0_mads_rcvd_unknown); > OSM_LOG(p_ctrl->p_log, OSM_LOG_ERROR, "ERR 3112: " > diff --git a/opensm/opensm/osm_trap_rcv.c b/opensm/opensm/osm_trap_rcv.c > index a5491e1..8a52f1a 100644 > --- a/opensm/opensm/osm_trap_rcv.c > +++ b/opensm/opensm/osm_trap_rcv.c > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2009 HNR Consulting. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -649,18 +650,34 @@ Exit: > } > > /********************************************************************** > - CURRENTLY WE ARE NOT CREATING TRAPS - SO THIS CALL IN AN ERROR > **********************************************************************/ > static void > trap_rcv_process_response(IN osm_sm_t * sm, > IN const osm_madw_t * const p_madw) > { > + uint8_t payload[sizeof(ib_mad_notice_attr_t)]; > + ib_smp_t *p_smp; > + ib_mad_notice_attr_t *p_ntci = (ib_mad_notice_attr_t *) payload; > + > + CL_ASSERT(p_madw); > + > + if (!osm_log_is_active(sm->p_log, OSM_LOG_VERBOSE)) > + return; > > OSM_LOG_ENTER(sm->p_log); > > - OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3808: " > - "This function is not supported yet\n"); > + if (p_madw->p_mad->mgmt_class != IB_MCLASS_SUBN_LID && > + p_madw->p_mad->mgmt_class != IB_MCLASS_SUBN_DIR) > + goto Exit; > > + p_smp = osm_madw_get_smp_ptr(p_madw); > + > + memset(payload, 0, sizeof(payload)); > + memcpy(payload, &p_smp->data, IB_SMP_DATA_SIZE); > + > + osm_dump_notice(sm->p_log, p_ntci, OSM_LOG_VERBOSE); > + > +Exit: > OSM_LOG_EXIT(sm->p_log); > } > From sashak at voltaire.com Fri Mar 6 09:32:55 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 19:32:55 +0200 Subject: [ofa-general] Re: [PATCH 2/2] libvendor/osm_vendor_mlx_dispatcher.c: Eliminate no longer needed osmv_mad_is_response In-Reply-To: <20090305122607.GC20341@comcast.net> References: <20090305122607.GC20341@comcast.net> Message-ID: <20090306173249.GB9673@sashak.voltaire.com> On 07:26 Thu 05 Mar , Hal Rosenstock wrote: > diff --git a/opensm/libvendor/osm_vendor_ibumad.c b/opensm/libvendor/osm_vendor_ibumad.c > index 2d53ff4..f393676 100644 > --- a/opensm/libvendor/osm_vendor_ibumad.c > +++ b/opensm/libvendor/osm_vendor_ibumad.c > @@ -113,7 +113,7 @@ static void clear_madw(osm_vendor_t * p_vend) > pthread_mutex_unlock(&p_vend->match_tbl_mutex); > OSM_LOG(p_vend->p_log, OSM_LOG_ERROR, "ERR 5401: " > "evicting entry %p (tid was 0x%" PRIx64 ")\n", > - old_m, old_tid); > + old_m, cl_ntoh64(old_tid)); > goto Exit; > } > } I think this change was applied alredy with some another patch. Sasha From hal.rosenstock at gmail.com Fri Mar 6 09:55:39 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 6 Mar 2009 12:55:39 -0500 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH 1/2] opensm: Handle trap repress on trap 144 generation In-Reply-To: <20090306173026.GA9673@sashak.voltaire.com> References: <20090305122433.GB20341@comcast.net> <20090306173026.GA9673@sashak.voltaire.com> Message-ID: Sasha, On Fri, Mar 6, 2009 at 12:30 PM, Sasha Khapyorsky wrote: > Hi Hal, > > On 07:24 Thu 05 Mar     , Hal Rosenstock wrote: >> >> Make trap generation sending be a transaction (expect a response) >> and handle incoming trap repress (notice attribute) > > I don't see how this works. See below. > >> ib_mad_is_response is now changed to better reflect responses >> (not just R bit in MAD header) >> >> Signed-off-by: Hal Rosenstock > > > >> @@ -3797,8 +3810,10 @@ static inline boolean_t OSM_API >>  ib_mad_is_response(IN const ib_mad_t * const p_mad) >>  { >>       CL_ASSERT(p_mad); >> -     return ((p_mad->method & IB_MAD_METHOD_RESP_MASK) == >> -             IB_MAD_METHOD_RESP_MASK); >> +     return (p_mad->method & IB_MAD_METHOD_RESP_MASK || >> +             p_mad->method == IB_MAD_METHOD_TRAP_REPRESS || >> +             (p_mad->mgmt_class == IB_MCLASS_BM && >> +              p_mad->attr_mod & IB_BM_ATTR_MOD_RESP)); > > Why to mix things? It could be subsequent patch but thought it best to include it. Do you want this separate ? > BR is not used anywhere in OpenSM. No, but someone might use ib_types.h to build BM. > - Show quoted text - >>  } >> >>  /* >> diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c >> index 2e851b2..1a46e50 100644 >> --- a/opensm/opensm/osm_req.c >> +++ b/opensm/opensm/osm_req.c >> @@ -246,6 +246,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) >> >>       madw->mad_addr.dest_lid = pi->master_sm_base_lid; >>       madw->mad_addr.addr_type.smi.source_lid = pi->base_lid; >> +     madw->resp_expected = TRUE; >>       madw->fail_msg = CL_DISP_MSGID_NONE; >> >>       smp = osm_madw_get_smp_ptr(madw); >> diff --git a/opensm/opensm/osm_sm_mad_ctrl.c b/opensm/opensm/osm_sm_mad_ctrl.c >> index 267ec85..a9124fa 100644 >> --- a/opensm/opensm/osm_sm_mad_ctrl.c >> +++ b/opensm/opensm/osm_sm_mad_ctrl.c >> @@ -2,6 +2,7 @@ >>   * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. >>   * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. >>   * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. >> + * Copyright (c) 2009 HNR Consulting. All rights reserved. >>   * >>   * This software is available to you under a choice of one of two >>   * licenses.  You may choose to be licensed under the terms of the GNU >> @@ -250,10 +251,12 @@ __osm_sm_mad_ctrl_process_get_resp(IN osm_sm_mad_ctrl_t * const p_ctrl, >>       case IB_MAD_ATTR_P_KEY_TABLE: >>               msg_id = OSM_MSG_MAD_PKEY; >>               break; >> +     case IB_MAD_ATTR_NOTICE: >> +             msg_id = OSM_MSG_MAD_NOTICE; >> +             break; > > This is GetResp processing, not TrapRepress. And here > osm_trap_rcv_process() is actually scheduled for execution. It isn't needed; I missed this when I cut the patch. > - Show quoted text - >> >>       case IB_MAD_ATTR_GUID_INFO: >>       case IB_MAD_ATTR_CLASS_PORT_INFO: >> -     case IB_MAD_ATTR_NOTICE: >>       case IB_MAD_ATTR_INFORM_INFO: >>       default: >>               cl_atomic_inc(&p_ctrl->p_stats->qp0_mads_rcvd_unknown); >> @@ -537,6 +540,57 @@ Exit: >>   * SEE ALSO >>   *********/ >> >> +/****f* opensm: SM/__osm_sm_mad_ctrl_process_trap_repress >> + * NAME >> + * __osm_sm_mad_ctrl_process_trap_repress >> + * >> + * DESCRIPTION >> + * This function handles method TrapRepress() for received MADs. >> + * >> + * SYNOPSIS >> + */ >> +static void >> +__osm_sm_mad_ctrl_process_trap_repress(IN osm_sm_mad_ctrl_t * const p_ctrl, >> +                                    IN osm_madw_t * p_madw) >> +{ >> +     ib_smp_t *p_smp; >> + >> +     OSM_LOG_ENTER(p_ctrl->p_log); >> + >> +     p_smp = osm_madw_get_smp_ptr(p_madw); >> + >> +     /* >> +        Note that attr_id (like the rest of the MAD) is in >> +        network byte order. >> +      */ >> +     switch (p_smp->attr_id) { >> +     case IB_MAD_ATTR_NOTICE: >> +             break; >> + >> +     default: >> +             cl_atomic_inc(&p_ctrl->p_stats->qp0_mads_rcvd_unknown); >> +             OSM_LOG(p_ctrl->p_log, OSM_LOG_ERROR, "ERR 3105: " >> +                     "Unsupported attribute = 0x%X\n", >> +                     cl_ntoh16(p_smp->attr_id)); >> +             osm_dump_dr_smp(p_ctrl->p_log, p_smp, OSM_LOG_ERROR); >> +             break; >> +     } >> + >> +     osm_mad_pool_put(p_ctrl->p_mad_pool, p_madw); >> + >> +     OSM_LOG_EXIT(p_ctrl->p_log); >> +} > > And this function which should process TrapRepress method does nothing, > right? Just some validation; It doesn't need to do anything (just retire the transaction). -- Hal >> + >> +/* >> + * PARAMETERS >> + * >> + * RETURN VALUES >> + * >> + * NOTES >> + * >> + * SEE ALSO >> + *********/ >> + >>  /****f* opensm: SM/__osm_sm_mad_ctrl_rcv_callback >>   * NAME >>   * __osm_sm_mad_ctrl_rcv_callback >> @@ -624,10 +678,14 @@ __osm_sm_mad_ctrl_rcv_callback(IN osm_madw_t * p_madw, > > This (__osm_sm_mad_ctrl_rcv_callback()) function has above > >        if (ib_smp_is_response(p_smp) || >            (p_smp->method == IB_MAD_METHOD_TRAP_REPRESS)) > > check. Following your ib_mad_is_response() modification it should be > simplified (probably in other places too). > > Sasha > - Show quoted text - >>               __osm_sm_mad_ctrl_process_set(p_ctrl, p_madw); >>               break; >> >> +     case IB_MAD_METHOD_TRAP_REPRESS: >> +             CL_ASSERT(p_req_madw == NULL); >> +             __osm_sm_mad_ctrl_process_trap_repress(p_ctrl, p_madw); >> +             break; >> + >>       case IB_MAD_METHOD_SEND: >>       case IB_MAD_METHOD_REPORT: >>       case IB_MAD_METHOD_REPORT_RESP: >> -     case IB_MAD_METHOD_TRAP_REPRESS: >>       default: >>               cl_atomic_inc(&p_ctrl->p_stats->qp0_mads_rcvd_unknown); >>               OSM_LOG(p_ctrl->p_log, OSM_LOG_ERROR, "ERR 3112: " >> diff --git a/opensm/opensm/osm_trap_rcv.c b/opensm/opensm/osm_trap_rcv.c >> index a5491e1..8a52f1a 100644 >> --- a/opensm/opensm/osm_trap_rcv.c >> +++ b/opensm/opensm/osm_trap_rcv.c >> @@ -2,6 +2,7 @@ >>   * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. >>   * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. >>   * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. >> + * Copyright (c) 2009 HNR Consulting. All rights reserved. >>   * >>   * This software is available to you under a choice of one of two >>   * licenses.  You may choose to be licensed under the terms of the GNU >> @@ -649,18 +650,34 @@ Exit: >>  } >> >>  /********************************************************************** >> - CURRENTLY WE ARE NOT CREATING TRAPS - SO THIS CALL IN AN ERROR >>  **********************************************************************/ >>  static void >>  trap_rcv_process_response(IN osm_sm_t * sm, >>                         IN const osm_madw_t * const p_madw) >>  { >> +     uint8_t payload[sizeof(ib_mad_notice_attr_t)]; >> +     ib_smp_t *p_smp; >> +     ib_mad_notice_attr_t *p_ntci = (ib_mad_notice_attr_t *) payload; >> + >> +     CL_ASSERT(p_madw); >> + >> +     if (!osm_log_is_active(sm->p_log, OSM_LOG_VERBOSE)) >> +             return; >> >>       OSM_LOG_ENTER(sm->p_log); >> >> -     OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3808: " >> -             "This function is not supported yet\n"); >> +     if (p_madw->p_mad->mgmt_class != IB_MCLASS_SUBN_LID && >> +         p_madw->p_mad->mgmt_class != IB_MCLASS_SUBN_DIR) >> +             goto Exit; >> >> +     p_smp = osm_madw_get_smp_ptr(p_madw); >> + >> +     memset(payload, 0, sizeof(payload)); >> +     memcpy(payload, &p_smp->data, IB_SMP_DATA_SIZE); >> + >> +     osm_dump_notice(sm->p_log, p_ntci, OSM_LOG_VERBOSE); >> + >> +Exit: >>       OSM_LOG_EXIT(sm->p_log); >>  } >> > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hal.rosenstock at gmail.com Fri Mar 6 09:56:28 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 6 Mar 2009 12:56:28 -0500 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH 2/2] libvendor/osm_vendor_mlx_dispatcher.c: Eliminate no longer needed osmv_mad_is_response In-Reply-To: <20090306173249.GB9673@sashak.voltaire.com> References: <20090305122607.GC20341@comcast.net> <20090306173249.GB9673@sashak.voltaire.com> Message-ID: On Fri, Mar 6, 2009 at 12:32 PM, Sasha Khapyorsky wrote: > On 07:26 Thu 05 Mar     , Hal Rosenstock wrote: > > >> diff --git a/opensm/libvendor/osm_vendor_ibumad.c b/opensm/libvendor/osm_vendor_ibumad.c >> index 2d53ff4..f393676 100644 >> --- a/opensm/libvendor/osm_vendor_ibumad.c >> +++ b/opensm/libvendor/osm_vendor_ibumad.c >> @@ -113,7 +113,7 @@ static void clear_madw(osm_vendor_t * p_vend) >>                       pthread_mutex_unlock(&p_vend->match_tbl_mutex); >>                       OSM_LOG(p_vend->p_log, OSM_LOG_ERROR, "ERR 5401: " >>                               "evicting entry %p (tid was 0x%" PRIx64 ")\n", >> -                             old_m, old_tid); >> +                             old_m, cl_ntoh64(old_tid)); >>                       goto Exit; >>               } >>       } > > I think this change was applied alredy with some another patch. Yes, this was mistakenly included. -- Hal > Sasha > - Show quoted text - > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hal.rosenstock at gmail.com Fri Mar 6 09:59:50 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 6 Mar 2009 12:59:50 -0500 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH 1/2] opensm: Handle trap repress on trap 144 generation In-Reply-To: <20090306173026.GA9673@sashak.voltaire.com> References: <20090305122433.GB20341@comcast.net> <20090306173026.GA9673@sashak.voltaire.com> Message-ID: Sasha, Missed this in my previous response: On Fri, Mar 6, 2009 at 12:30 PM, Sasha Khapyorsky wrote: >> @@ -624,10 +678,14 @@ __osm_sm_mad_ctrl_rcv_callback(IN osm_madw_t * p_madw, > > This (__osm_sm_mad_ctrl_rcv_callback()) function has above > >        if (ib_smp_is_response(p_smp) || >            (p_smp->method == IB_MAD_METHOD_TRAP_REPRESS)) > > check. Following your ib_mad_is_response() modification it should be > simplified (probably in other places too). Only found a couple of other places: libvendor/osm_vendor_mtl.c and osm_vendor_ts.c. -- Hal > Sasha From sean.hefty at intel.com Fri Mar 6 10:03:04 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Fri, 6 Mar 2009 10:03:04 -0800 Subject: [ofa-general] RE: [PATCH] [ib-diag] ibstat: port_dump is called too many times In-Reply-To: <20090306124547.GV6258@sashak.voltaire.com> References: <40F59C0D0E0E47F99E3D21458877BF31@amr.corp.intel.com> <20090306124547.GV6258@sashak.voltaire.com> Message-ID: >> - for (portnum = 0; portnum <= ca.numports; portnum++) >> + for (portnum = 0; portnum < ca.numports; portnum++) >> port_dump(ca.ports[portnum], 0); > >As expected on linux it shows only first port of two ports CA. Correct - port_dump checks for NULL ports and returns. This patch does break Linux. >Likely we have libibumad implemented differently in this part. In libibumad, is umad_ca_t:ports[0] ever not NULL? - Sean From weiny2 at llnl.gov Fri Mar 6 10:04:05 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 6 Mar 2009 10:04:05 -0800 Subject: [ofa-general] Re: [PATCH 2/10] infiniband-diags: Convert ibaddr to "new" ibmad interface In-Reply-To: <20090302160659.0807fb24.weiny2@llnl.gov> References: <20090219190528.11c080f8.weiny2@llnl.gov> <20090301080804.GH3936@sashak.voltaire.com> <20090302160659.0807fb24.weiny2@llnl.gov> Message-ID: <20090306100405.8dd79ac8.weiny2@llnl.gov> On Mon, 2 Mar 2009 16:06:59 -0800 Ira Weiny wrote: > On Sun, 1 Mar 2009 10:08:04 +0200 > Sasha Khapyorsky wrote: > > > On 19:05 Thu 19 Feb , Ira Weiny wrote: > > [snip] > > > > > > > - madrpc_init(ibd_ca, ibd_ca_port, mgmt_classes, 3); > > > + srcport = mad_rpc_open_port(ibd_ca, ibd_ca_port, mgmt_classes, 3); > > > + if (!srcport) > > > + IBERROR("Failed to open '%s' port '%d'", ibd_ca, ibd_ca_port); > > > > Here and in almost all other infiniband-diags patches: > > > > madrpc_init() in case of failures calls IBPANIC() which prints error > > message and calls exit(). Unlike this mad_rpc_open_port() sets errno and > > returns NULL, IBERROR() just prints error message - the program will > > continue execution... > > Good catch I will update with a proper exit call. > Actually I think I was right the first time. IBERROR is a macro which calls iberror. iberror calls exit(-1) at the end. ibdiag_common.h: #define IBERROR(fmt, ...) iberror(__FUNCTION__, fmt, ## __VA_ARGS__) ibdiag_common.c: void iberror(const char *fn, char *msg, ...) { ... exit(-1); } Is this not correct? We can change the behavior of IBERROR but I think we have enough to deal with right now. Ira > [snip] From hnrose at comcast.net Fri Mar 6 10:20:53 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Fri, 6 Mar 2009 13:20:53 -0500 Subject: [ofa-general] ***SPAM*** [PATCHv2 2/2] libvendor/osm_vendor_mlx_dispatcher.c: Eliminate no longer needed osmv_mad_is_response Message-ID: <20090306182053.GB1706@comcast.net> Use ib_mad_is_response (which now supports trap repress) Signed-off-by: Hal Rosenstock --- Changes from v1: Eliminated mistakenly included osm_vendor_ibumad.c change diff --git a/opensm/include/vendor/osm_vendor_mlx_svc.h b/opensm/include/vendor/osm_vendor_mlx_svc.h index f23a77d..43a83e3 100644 --- a/opensm/include/vendor/osm_vendor_mlx_svc.h +++ b/opensm/include/vendor/osm_vendor_mlx_svc.h @@ -50,12 +50,6 @@ #endif /* __cplusplus */ BEGIN_C_DECLS -inline static boolean_t osmv_mad_is_response(IN const ib_mad_t * p_mad) -{ - return (ib_mad_is_response(p_mad) || - (p_mad->method == IB_MAD_METHOD_TRAP_REPRESS)); -} - inline static uint8_t osmv_invert_method(IN uint8_t req_method) { switch (req_method) { diff --git a/opensm/libvendor/osm_vendor_mlx_dispatcher.c b/opensm/libvendor/osm_vendor_mlx_dispatcher.c index d476382..1954547 100644 --- a/opensm/libvendor/osm_vendor_mlx_dispatcher.c +++ b/opensm/libvendor/osm_vendor_mlx_dispatcher.c @@ -193,7 +193,7 @@ __osmv_dispatch_route(IN osm_bind_handle_t h_bind, { ib_api_status_t ret; osmv_bind_obj_t *p_bo = (osmv_bind_obj_t *) h_bind; - boolean_t is_resp = osmv_mad_is_response(p_mad); + boolean_t is_resp = ib_mad_is_response(p_mad); boolean_t is_txn; uint64_t key = cl_ntoh64(p_mad->trans_id); From hnrose at comcast.net Fri Mar 6 10:19:48 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Fri, 6 Mar 2009 13:19:48 -0500 Subject: [ofa-general] [PATCH v2 1/2] opensm: Handle trap repress on trap 144 generation Message-ID: <20090306181948.GA1706@comcast.net> Make trap generation sending be a transaction (expect a response) and handle incoming trap repress (notice attribute) ib_mad_is_response is now changed to better reflect responses (not just R bit in MAD header) Signed-off-by: Hal Rosenstock --- Changes from v1: Removed mistaken IB_MAD_ATTR_NOTICE handling in GetResp routine Removed superfluous check of trap repress as ib_smp_is_response is now sufficient by itself Added similar modification to several vendor layers (not compiled) diff --git a/opensm/include/iba/ib_types.h b/opensm/include/iba/ib_types.h index 205d160..06f3e75 100644 --- a/opensm/include/iba/ib_types.h +++ b/opensm/include/iba/ib_types.h @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -3763,8 +3764,8 @@ static inline boolean_t OSM_API ib_mad_is_response(IN const ib_mad_t * const p_mad) { CL_ASSERT(p_mad); - return ((p_mad->method & IB_MAD_METHOD_RESP_MASK) == - IB_MAD_METHOD_RESP_MASK); + return (p_mad->method & IB_MAD_METHOD_RESP_MASK || + p_mad->method == IB_MAD_METHOD_TRAP_REPRESS); } /* diff --git a/opensm/libvendor/osm_vendor_mtl.c b/opensm/libvendor/osm_vendor_mtl.c index 5fb05ef..6fc607e 100644 --- a/opensm/libvendor/osm_vendor_mtl.c +++ b/opensm/libvendor/osm_vendor_mtl.c @@ -66,8 +66,7 @@ uint64_t __osm_set_wrid_by_p_madw(IN osm_madw_t * p_madw) memcpy(&wrid, &p_madw, sizeof(osm_madw_t *)); wrid = (wrid << 1) | - ib_mad_is_response(p_madw->p_mad) | - (p_madw->p_mad->method == IB_MAD_METHOD_TRAP_REPRESS); + ib_mad_is_response(p_madw->p_mad); return wrid; } diff --git a/opensm/libvendor/osm_vendor_ts.c b/opensm/libvendor/osm_vendor_ts.c index 710d06f..2e24881 100644 --- a/opensm/libvendor/osm_vendor_ts.c +++ b/opensm/libvendor/osm_vendor_ts.c @@ -61,8 +61,7 @@ uint64_t __osm_set_wrid_by_p_madw(IN osm_madw_t * p_madw) memcpy(&wrid, &p_madw, sizeof(osm_madw_t *)); wrid = (wrid << 1) | - ib_mad_is_response(p_madw->p_mad) | - (p_madw->p_mad->method == IB_MAD_METHOD_TRAP_REPRESS); + ib_mad_is_response(p_madw->p_mad); return wrid; } diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c index 2e851b2..1a46e50 100644 --- a/opensm/opensm/osm_req.c +++ b/opensm/opensm/osm_req.c @@ -246,6 +246,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) madw->mad_addr.dest_lid = pi->master_sm_base_lid; madw->mad_addr.addr_type.smi.source_lid = pi->base_lid; + madw->resp_expected = TRUE; madw->fail_msg = CL_DISP_MSGID_NONE; smp = osm_madw_get_smp_ptr(madw); diff --git a/opensm/opensm/osm_sm_mad_ctrl.c b/opensm/opensm/osm_sm_mad_ctrl.c index 267ec85..5bea57e 100644 --- a/opensm/opensm/osm_sm_mad_ctrl.c +++ b/opensm/opensm/osm_sm_mad_ctrl.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -537,6 +538,57 @@ Exit: * SEE ALSO *********/ +/****f* opensm: SM/__osm_sm_mad_ctrl_process_trap_repress + * NAME + * __osm_sm_mad_ctrl_process_trap_repress + * + * DESCRIPTION + * This function handles method TrapRepress() for received MADs. + * + * SYNOPSIS + */ +static void +__osm_sm_mad_ctrl_process_trap_repress(IN osm_sm_mad_ctrl_t * const p_ctrl, + IN osm_madw_t * p_madw) +{ + ib_smp_t *p_smp; + + OSM_LOG_ENTER(p_ctrl->p_log); + + p_smp = osm_madw_get_smp_ptr(p_madw); + + /* + Note that attr_id (like the rest of the MAD) is in + network byte order. + */ + switch (p_smp->attr_id) { + case IB_MAD_ATTR_NOTICE: + break; + + default: + cl_atomic_inc(&p_ctrl->p_stats->qp0_mads_rcvd_unknown); + OSM_LOG(p_ctrl->p_log, OSM_LOG_ERROR, "ERR 3105: " + "Unsupported attribute = 0x%X\n", + cl_ntoh16(p_smp->attr_id)); + osm_dump_dr_smp(p_ctrl->p_log, p_smp, OSM_LOG_ERROR); + break; + } + + osm_mad_pool_put(p_ctrl->p_mad_pool, p_madw); + + OSM_LOG_EXIT(p_ctrl->p_log); +} + +/* + * PARAMETERS + * + * RETURN VALUES + * + * NOTES + * + * SEE ALSO + *********/ + /****f* opensm: SM/__osm_sm_mad_ctrl_rcv_callback * NAME * __osm_sm_mad_ctrl_rcv_callback @@ -577,8 +629,7 @@ __osm_sm_mad_ctrl_rcv_callback(IN osm_madw_t * p_madw, osm_dump_dr_smp(p_ctrl->p_log, p_smp, OSM_LOG_DEBUG); /* retire the mad or put it back */ - if (ib_smp_is_response(p_smp) || - (p_smp->method == IB_MAD_METHOD_TRAP_REPRESS)) { + if (ib_smp_is_response(p_smp)) { CL_ASSERT(p_madw->resp_expected == FALSE); __osm_sm_mad_ctrl_retire_trans_mad(p_ctrl, p_madw); } else if (p_madw->resp_expected == TRUE) @@ -624,10 +675,14 @@ __osm_sm_mad_ctrl_rcv_callback(IN osm_madw_t * p_madw, __osm_sm_mad_ctrl_process_set(p_ctrl, p_madw); break; + case IB_MAD_METHOD_TRAP_REPRESS: + CL_ASSERT(p_req_madw == NULL); + __osm_sm_mad_ctrl_process_trap_repress(p_ctrl, p_madw); + break; + case IB_MAD_METHOD_SEND: case IB_MAD_METHOD_REPORT: case IB_MAD_METHOD_REPORT_RESP: - case IB_MAD_METHOD_TRAP_REPRESS: default: cl_atomic_inc(&p_ctrl->p_stats->qp0_mads_rcvd_unknown); OSM_LOG(p_ctrl->p_log, OSM_LOG_ERROR, "ERR 3112: " diff --git a/opensm/opensm/osm_trap_rcv.c b/opensm/opensm/osm_trap_rcv.c index 6639246..4576a8b 100644 --- a/opensm/opensm/osm_trap_rcv.c +++ b/opensm/opensm/osm_trap_rcv.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -649,18 +650,34 @@ Exit: } /********************************************************************** - CURRENTLY WE ARE NOT CREATING TRAPS - SO THIS CALL IN AN ERROR **********************************************************************/ static void trap_rcv_process_response(IN osm_sm_t * sm, IN const osm_madw_t * const p_madw) { + uint8_t payload[sizeof(ib_mad_notice_attr_t)]; + ib_smp_t *p_smp; + ib_mad_notice_attr_t *p_ntci = (ib_mad_notice_attr_t *) payload; + + CL_ASSERT(p_madw); + + if (!osm_log_is_active(sm->p_log, OSM_LOG_VERBOSE)) + return; OSM_LOG_ENTER(sm->p_log); - OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3808: " - "This function is not supported yet\n"); + if (p_madw->p_mad->mgmt_class != IB_MCLASS_SUBN_LID && + p_madw->p_mad->mgmt_class != IB_MCLASS_SUBN_DIR) + goto Exit; + p_smp = osm_madw_get_smp_ptr(p_madw); + + memset(payload, 0, sizeof(payload)); + memcpy(payload, &p_smp->data, IB_SMP_DATA_SIZE); + + osm_dump_notice(sm->p_log, p_ntci, OSM_LOG_VERBOSE); + +Exit: OSM_LOG_EXIT(sm->p_log); } From sean.hefty at intel.com Fri Mar 6 10:45:30 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Fri, 6 Mar 2009 10:45:30 -0800 Subject: [ofa-general] RE: complib and cl_nodenamemap In-Reply-To: <20090306131701.GW6258@sashak.voltaire.com> References: <49E9FBB3AEB64CEDA3577778C69E6392@amr.corp.intel.com> <20090306131701.GW6258@sashak.voltaire.com> Message-ID: >Basically yes. But we need to find some acceptable solution first which >will be better than how it is implemented now. One possibility is to link the calls directly. This is the same as calls in ibdiag_common.c. Functions like ibdiag_process_opts() are used in more places than cl_nodenamemap, but they are still not part of any library. This is the simplest solution for now. An alternative is to abstract the needed functionality in such a way that it's usable not just by the IB management code, but other potential users as well. This requires more engineering effort. My personal preference is to link the calls in directly, then decide later what, if anything, should be done with code shared between all the IB management applications. - Sean From weiny2 at llnl.gov Fri Mar 6 10:48:08 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 6 Mar 2009 10:48:08 -0800 Subject: [ofa-general] Re: [PATCH 2/10] infiniband-diags: Convert ibaddr to "new" ibmad interface In-Reply-To: <20090306085002.GF6258@sashak.voltaire.com> References: <20090219190528.11c080f8.weiny2@llnl.gov> <20090301080804.GH3936@sashak.voltaire.com> <20090302160659.0807fb24.weiny2@llnl.gov> <20090306085002.GF6258@sashak.voltaire.com> Message-ID: <20090306104808.b3ba7f8a.weiny2@llnl.gov> On Fri, 6 Mar 2009 10:50:18 +0200 Sasha Khapyorsky wrote: > On 16:06 Mon 02 Mar , Ira Weiny wrote: > > > > diff --git a/infiniband-diags/src/ibdiag_common.c b/infiniband-diags/src/ibdiag_common.c > > > > index 5f2472d..609df69 100644 > > > > --- a/infiniband-diags/src/ibdiag_common.c > > > > +++ b/infiniband-diags/src/ibdiag_common.c > > > > @@ -179,7 +179,8 @@ static int process_opt(int ch, char *optarg) > > > > ibd_timeout = val; > > > > break; > > > > case 's': > > > > - if (ib_resolve_portid_str(&sm_portid, optarg, IB_DEST_LID, 0) < 0) > > > > + if (ib_resolve_portid_str_via(&sm_portid, optarg, IB_DEST_LID, > > > > + 0, NULL) < 0) > > > > > > This may be not correct (not introduced by those patches) - then host is > > > connected to multiple subnets we need to resolve SM LID on requested > > > one, eg. using srcport... > > > > > > > Would you accept adding srcport to ibdiag_process_opts? Or would you prefer > > something more generic? > > More generic would be better :) > > BTW it could be not so easy - when ibdiag_process_opts() is running > srcport is not initialized yet. > Actually, you don't need a srcport for this function. When "resolving" via LID nothing goes on the wire. It only fills in ib_portid_t struct. The "resolve" name is a bit misleading in this case. Because "-s" always specifies a there is nothing to actually "resolve". I have added a comment reflecting this. New patch below, Ira From: Ira Weiny Date: Thu, 19 Feb 2009 14:47:05 -0800 Subject: [PATCH] infiniband-diags: Convert ibaddr to "new" ibmad interface Signed-off-by: Ira Weiny --- infiniband-diags/src/ibaddr.c | 17 ++++++++++++----- infiniband-diags/src/ibdiag_common.c | 4 +++- 2 files changed, 15 insertions(+), 6 deletions(-) diff --git a/infiniband-diags/src/ibaddr.c b/infiniband-diags/src/ibaddr.c index 9098699..bb22be9 100644 --- a/infiniband-diags/src/ibaddr.c +++ b/infiniband-diags/src/ibaddr.c @@ -45,6 +45,8 @@ #include "ibdiag_common.h" +struct ibmad_port *srcport; + static int ib_resolve_addr(ib_portid_t *portid, int portnum, int show_lid, int show_gid) { @@ -55,10 +57,10 @@ ib_resolve_addr(ib_portid_t *portid, int portnum, int show_lid, int show_gid) ibmad_gid_t gid; int lmc; - if (!smp_query(nodeinfo, portid, IB_ATTR_NODE_INFO, 0, 0)) + if (!smp_query_via(nodeinfo, portid, IB_ATTR_NODE_INFO, 0, 0, srcport)) return -1; - if (!smp_query(portinfo, portid, IB_ATTR_PORT_INFO, portnum, 0)) + if (!smp_query_via(portinfo, portid, IB_ATTR_PORT_INFO, portnum, 0, srcport)) return -1; mad_decode_field(portinfo, IB_PORT_LID_F, &portid->lid); @@ -137,17 +139,22 @@ int main(int argc, char **argv) if (!show_lid && !show_gid) show_lid = show_gid = 1; - madrpc_init(ibd_ca, ibd_ca_port, mgmt_classes, 3); + srcport = mad_rpc_open_port(ibd_ca, ibd_ca_port, mgmt_classes, 3); + if (!srcport) + IBERROR("Failed to open '%s' port '%d'", ibd_ca, ibd_ca_port); if (argc) { - if (ib_resolve_portid_str(&portid, argv[0], ibd_dest_type, ibd_sm_id) < 0) + if (ib_resolve_portid_str_via(&portid, argv[0], ibd_dest_type, + ibd_sm_id, srcport) < 0) IBERROR("can't resolve destination port %s", argv[0]); } else { - if (ib_resolve_self(&portid, &port, 0) < 0) + if (ib_resolve_self_via(&portid, &port, 0, srcport) < 0) IBERROR("can't resolve self port %s", argv[0]); } if (ib_resolve_addr(&portid, port, show_lid, show_gid) < 0) IBERROR("can't resolve requested address"); + + mad_rpc_close_port(srcport); exit(0); } diff --git a/infiniband-diags/src/ibdiag_common.c b/infiniband-diags/src/ibdiag_common.c index 5f2472d..c0421f6 100644 --- a/infiniband-diags/src/ibdiag_common.c +++ b/infiniband-diags/src/ibdiag_common.c @@ -179,7 +179,9 @@ static int process_opt(int ch, char *optarg) ibd_timeout = val; break; case 's': - if (ib_resolve_portid_str(&sm_portid, optarg, IB_DEST_LID, 0) < 0) + /* srcport is not required when resolving via IB_DEST_LID */ + if (ib_resolve_portid_str_via(&sm_portid, optarg, IB_DEST_LID, + 0, NULL) < 0) IBERROR("cannot resolve SM destination port %s", optarg); ibd_sm_id = &sm_portid; break; -- 1.5.4.5 From weiny2 at llnl.gov Fri Mar 6 10:56:48 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 6 Mar 2009 10:56:48 -0800 Subject: [ofa-general] RE: complib and cl_nodenamemap In-Reply-To: References: <49E9FBB3AEB64CEDA3577778C69E6392@amr.corp.intel.com> <20090306131701.GW6258@sashak.voltaire.com> Message-ID: <20090306105648.ff23413b.weiny2@llnl.gov> On Fri, 6 Mar 2009 10:45:30 -0800 "Sean Hefty" wrote: > >Basically yes. But we need to find some acceptable solution first which > >will be better than how it is implemented now. > > One possibility is to link the calls directly. This is the same as calls in > ibdiag_common.c. Functions like ibdiag_process_opts() are used in more places > than cl_nodenamemap, but they are still not part of any library. This is the > simplest solution for now. Do you mean via a static library? Since the calls are used in 2 packages opensm and infiniband-diags, you will have to have a library somewhere to preserve building the packages separately; as RedHat and we do. > > An alternative is to abstract the needed functionality in such a way that it's > usable not just by the IB management code, but other potential users as well. > This requires more engineering effort. > > My personal preference is to link the calls in directly, then decide later what, > if anything, should be done with code shared between all the IB management > applications. > Shared code, across the different packages, must be in a library somewhere. Since ibcommon was being deprecated at the time (and is gone now) complib was the "logical" place to put this code. :-( Ira From swise at opengridcomputing.com Fri Mar 6 11:48:37 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Fri, 06 Mar 2009 13:48:37 -0600 Subject: [ofa-general] tcp/rdma port unification patch in librdmacm Message-ID: <49B17E15.4050406@opengridcomputing.com> Hey Sean, It seems like kernel RDMA protocols are all getting IANA-assigned port numbers that alleviates the need for TCP/RDMA port unification. NFSRDMA and RDS are two. But user applications like mvapich2 and other mpis often let the stack choose a local port number on-which to bind/listen. Would you reconsider moving the tcp/rdma port unification changes up into librdmacm? The benefit is that this will resolve the issue in a manner that can flow into the distros and not affect the linux kernel. So librdmacm could, if configured to support this, just allocate and bind a user socket to reserve the TCP port on behalf of the rdma application. The code would be similar to the kernel patch... Thoughts? Thanks, Steve. From sashak at voltaire.com Fri Mar 6 11:52:27 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 21:52:27 +0200 Subject: [ofa-general] Re: [PATCH] [ib-diag] ibstat: port_dump is called too many times In-Reply-To: References: <40F59C0D0E0E47F99E3D21458877BF31@amr.corp.intel.com> <20090306124547.GV6258@sashak.voltaire.com> Message-ID: <20090306195220.GD9673@sashak.voltaire.com> On 10:03 Fri 06 Mar , Sean Hefty wrote: > > In libibumad, is umad_ca_t:ports[0] ever not NULL? Yes, when running on switch. Sasha From sean.hefty at intel.com Fri Mar 6 11:49:15 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Fri, 6 Mar 2009 11:49:15 -0800 Subject: [ofa-general] RE: complib and cl_nodenamemap In-Reply-To: <20090306105648.ff23413b.weiny2@llnl.gov> References: <49E9FBB3AEB64CEDA3577778C69E6392@amr.corp.intel.com> <20090306131701.GW6258@sashak.voltaire.com> <20090306105648.ff23413b.weiny2@llnl.gov> Message-ID: <4260917061B84A199895736482131117@amr.corp.intel.com> >Do you mean via a static library? Since the calls are used in 2 packages >opensm and infiniband-diags, you will have to have a library somewhere to >preserve building the packages separately; as RedHat and we do. ibdiags can build and use it like ibdiag_common.c. (Add to libcommon, which gets linked in statically to each diag?) opensm can reference it as one of its sources files. - Sean From sashak at voltaire.com Fri Mar 6 11:59:15 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 21:59:15 +0200 Subject: [ofa-general] Re: [PATCH 2/10] infiniband-diags: Convert ibaddr to "new" ibmad interface In-Reply-To: <20090306100405.8dd79ac8.weiny2@llnl.gov> References: <20090219190528.11c080f8.weiny2@llnl.gov> <20090301080804.GH3936@sashak.voltaire.com> <20090302160659.0807fb24.weiny2@llnl.gov> <20090306100405.8dd79ac8.weiny2@llnl.gov> Message-ID: <20090306195915.GE9673@sashak.voltaire.com> On 10:04 Fri 06 Mar , Ira Weiny wrote: > > Actually I think I was right the first time. IBERROR is a macro which calls > iberror. iberror calls exit(-1) at the end. > > ibdiag_common.h: > #define IBERROR(fmt, ...) iberror(__FUNCTION__, fmt, ## __VA_ARGS__) > > ibdiag_common.c: > void iberror(const char *fn, char *msg, ...) > { > ... > exit(-1); > } > > Is this not correct? Yes. Seems I missed IBERROR() with IBWARN(). Sasha From sean.hefty at intel.com Fri Mar 6 11:58:02 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Fri, 6 Mar 2009 11:58:02 -0800 Subject: [ofa-general] RE: tcp/rdma port unification patch in librdmacm In-Reply-To: <49B17E15.4050406@opengridcomputing.com> References: <49B17E15.4050406@opengridcomputing.com> Message-ID: <811553D6D34A4E6F9AB686CA2E441D35@amr.corp.intel.com> >Would you reconsider moving the tcp/rdma port unification changes up >into librdmacm? The benefit is that this will resolve the issue in a >manner that can flow into the distros and not affect the linux kernel. >So librdmacm could, if configured to support this, just allocate and >bind a user socket to reserve the TCP port on behalf of the rdma >application. The code would be similar to the kernel patch... > >Thoughts? This is what I ended up needing to do for Windows (but for other reasons), so I'm not against adding this to the library. I'm undecided on the best way to expose this. Always enabled? Enabled during the build? Enabled through a new API? Other? - Sean From sashak at voltaire.com Fri Mar 6 12:07:02 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 22:07:02 +0200 Subject: [ofa-general] RE: complib and cl_nodenamemap In-Reply-To: <4260917061B84A199895736482131117@amr.corp.intel.com> References: <49E9FBB3AEB64CEDA3577778C69E6392@amr.corp.intel.com> <20090306131701.GW6258@sashak.voltaire.com> <20090306105648.ff23413b.weiny2@llnl.gov> <4260917061B84A199895736482131117@amr.corp.intel.com> Message-ID: <20090306200702.GF9673@sashak.voltaire.com> On 11:49 Fri 06 Mar , Sean Hefty wrote: > >Do you mean via a static library? Since the calls are used in 2 packages > >opensm and infiniband-diags, you will have to have a library somewhere to > >preserve building the packages separately; as RedHat and we do. > > ibdiags can build and use it like ibdiag_common.c. (Add to libcommon, which > gets linked in statically to each diag?) opensm can reference it as one of its > sources files. We clearly don't want to introduce OpenSM -> infiniband-diags dependency - OpenSM now is distributable as separate tarball. Sasha From swise at opengridcomputing.com Fri Mar 6 12:36:58 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Fri, 06 Mar 2009 14:36:58 -0600 Subject: [ofa-general] Re: tcp/rdma port unification patch in librdmacm In-Reply-To: <811553D6D34A4E6F9AB686CA2E441D35@amr.corp.intel.com> References: <49B17E15.4050406@opengridcomputing.com> <811553D6D34A4E6F9AB686CA2E441D35@amr.corp.intel.com> Message-ID: <49B1896A.2080904@opengridcomputing.com> Sean Hefty wrote: >> Would you reconsider moving the tcp/rdma port unification changes up >> into librdmacm? The benefit is that this will resolve the issue in a >> manner that can flow into the distros and not affect the linux kernel. >> So librdmacm could, if configured to support this, just allocate and >> bind a user socket to reserve the TCP port on behalf of the rdma >> application. The code would be similar to the kernel patch... >> >> Thoughts? >> > > This is what I ended up needing to do for Windows (but for other reasons), so > I'm not against adding this to the library. > I did this too for Chelsio's NetDirect provider... > I'm undecided on the best way to expose this. Always enabled? Enabled during > the build? Enabled through a new API? Other? > > I think it should be enabled by default to resolve the issue once and for all. Steve. From sashak at voltaire.com Fri Mar 6 12:41:23 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 22:41:23 +0200 Subject: [ofa-general] Re: [PATCH 1/2] opensm: Handle trap repress on trap 144 generation In-Reply-To: References: <20090305122433.GB20341@comcast.net> <20090306173026.GA9673@sashak.voltaire.com> Message-ID: <20090306204123.GG9673@sashak.voltaire.com> On 12:55 Fri 06 Mar , Hal Rosenstock wrote: > > It could be subsequent patch but thought it best to include it. Do you > want this separate ? Without clear usage case I would prefer to not have it at all - finally it is on a fast path. > > > BR is not used anywhere in OpenSM. > > No, but someone might use ib_types.h to build BM. We cannot know what will be needed then - this someone will need to care anyway. > > And this function which should process TrapRepress method does nothing, > > right? > > Just some validation; It doesn't need to do anything (just retire the > transaction). Then I don't understand - how is trap 144 repress handled? And why those changes in trap_rcv_process_response() were needed? Sasha From hal.rosenstock at gmail.com Fri Mar 6 12:51:21 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 6 Mar 2009 15:51:21 -0500 Subject: [ofa-general] Re: [PATCH 1/2] opensm: Handle trap repress on trap 144 generation In-Reply-To: <20090306204123.GG9673@sashak.voltaire.com> References: <20090305122433.GB20341@comcast.net> <20090306173026.GA9673@sashak.voltaire.com> <20090306204123.GG9673@sashak.voltaire.com> Message-ID: On Fri, Mar 6, 2009 at 3:41 PM, Sasha Khapyorsky wrote: > On 12:55 Fri 06 Mar     , Hal Rosenstock wrote: >> >> It could be subsequent patch but thought it best to include it. Do you >> want this separate ? > > Without clear usage case I would prefer to not have it at all - finally > it is on a fast path. It's the same as what is done in the kernel for MAD response but I removed it because I could see you were against this. >> >> > BR is not used anywhere in OpenSM. >> >> No, but someone might use ib_types.h to build BM. > > We cannot know what will be needed then - this someone will need to care > anyway. You could say that about a lot of things accepted which aren't fully integrated. >> > And this function which should process TrapRepress method does nothing, >> > right? >> >> Just some validation; It doesn't need to do anything (just retire the >> transaction). > > Then I don't understand - how is trap 144 repress handled? And why > those changes in trap_rcv_process_response() were needed? Perhaps it's being overly pendantic. I can revise the patch to not have the repress get this far. -- Hal > Sasha > From sean.hefty at intel.com Fri Mar 6 12:54:23 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Fri, 6 Mar 2009 12:54:23 -0800 Subject: [ofa-general] RE: complib and cl_nodenamemap In-Reply-To: <20090306200702.GF9673@sashak.voltaire.com> References: <49E9FBB3AEB64CEDA3577778C69E6392@amr.corp.intel.com> <20090306131701.GW6258@sashak.voltaire.com> <20090306105648.ff23413b.weiny2@llnl.gov> <4260917061B84A199895736482131117@amr.corp.intel.com> <20090306200702.GF9673@sashak.voltaire.com> Message-ID: >We clearly don't want to introduce OpenSM -> infiniband-diags dependency >- OpenSM now is distributable as separate tarball. I'm not suggesting adding a dependency. I'm saying: 1. complib should be completely independent of Infiniband management on Linux. Having it tightly coupled with the Linux IB management subnet manager is a mistake that should be fixed. 2. cl_nodenamemap is an IB management application specific use of more generic complib and C-runtime libraries. A way should be found to share the nodenamemap source code between those apps that does not involve complib. If we do not want to move nodenamemap into a new library, then each package can have its own copy. Eventually the ib-diags would not need to depend on opensm being installed. From hal.rosenstock at gmail.com Fri Mar 6 13:05:39 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 6 Mar 2009 16:05:39 -0500 Subject: [ofa-general] Re: [PATCH 1/2] opensm: Handle trap repress on trap 144 generation In-Reply-To: References: <20090305122433.GB20341@comcast.net> <20090306173026.GA9673@sashak.voltaire.com> <20090306204123.GG9673@sashak.voltaire.com> Message-ID: On Fri, Mar 6, 2009 at 3:51 PM, Hal Rosenstock wrote: > On Fri, Mar 6, 2009 at 3:41 PM, Sasha Khapyorsky wrote: >> On 12:55 Fri 06 Mar     , Hal Rosenstock wrote: >>> >>> It could be subsequent patch but thought it best to include it. Do you >>> want this separate ? >> >> Without clear usage case I would prefer to not have it at all - finally >> it is on a fast path. > > It's the same as what is done in the kernel for MAD response but I > removed it because I could see you were against this. > >>> >>> > BR is not used anywhere in OpenSM. >>> >>> No, but someone might use ib_types.h to build BM. >> >> We cannot know what will be needed then - this someone will need to care >> anyway. > > You could say that about a lot of things accepted which aren't fully integrated. > >>> > And this function which should process TrapRepress method does nothing, >>> > right? >>> >>> Just some validation; It doesn't need to do anything (just retire the >>> transaction). >> >> Then I don't understand - how is trap 144 repress handled? And why >> those changes in trap_rcv_process_response() were needed? > > Perhaps it's being overly pendantic. I can revise the patch to not > have the repress get this far. I misread your comment. trap_rcv needs to cause the repress for the trap to be generated. -- Hal > -- Hal > >> Sasha >> > From weiny2 at llnl.gov Fri Mar 6 13:12:19 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 6 Mar 2009 13:12:19 -0800 Subject: [ofa-general] RE: complib and cl_nodenamemap In-Reply-To: References: <49E9FBB3AEB64CEDA3577778C69E6392@amr.corp.intel.com> <20090306131701.GW6258@sashak.voltaire.com> <20090306105648.ff23413b.weiny2@llnl.gov> <4260917061B84A199895736482131117@amr.corp.intel.com> <20090306200702.GF9673@sashak.voltaire.com> Message-ID: <20090306131219.7322d26a.weiny2@llnl.gov> On Fri, 6 Mar 2009 12:54:23 -0800 "Sean Hefty" wrote: > >We clearly don't want to introduce OpenSM -> infiniband-diags dependency > >- OpenSM now is distributable as separate tarball. > > I'm not suggesting adding a dependency. I'm saying: > > 1. complib should be completely independent of Infiniband management on Linux. > Having it tightly coupled with the Linux IB management subnet manager is a > mistake that should be fixed. Ah! ok. > > 2. cl_nodenamemap is an IB management application specific use of more generic > complib and C-runtime libraries. A way should be found to share the nodenamemap > source code between those apps that does not involve complib. Ok, but this sounds like we are going back to ibcommon even if by a different name. > > If we do not want to move nodenamemap into a new library, then each package can > have its own copy. Eventually the ib-diags would not need to depend on opensm > being installed. > The idea was to make sure that any formatting changes to the config file would be consistent between opensm and the diags. So far this has not happened. However, IMHO I think it would be a step backward to have the code in 2 places. Ira From hal.rosenstock at gmail.com Fri Mar 6 13:12:42 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 6 Mar 2009 16:12:42 -0500 Subject: [ofa-general] Re: [PATCH 1/2] opensm: Handle trap repress on trap 144 generation In-Reply-To: References: <20090305122433.GB20341@comcast.net> <20090306173026.GA9673@sashak.voltaire.com> <20090306204123.GG9673@sashak.voltaire.com> Message-ID: On Fri, Mar 6, 2009 at 4:05 PM, Hal Rosenstock wrote: > - Show quoted text - > On Fri, Mar 6, 2009 at 3:51 PM, Hal Rosenstock wrote: >> On Fri, Mar 6, 2009 at 3:41 PM, Sasha Khapyorsky wrote: >>> On 12:55 Fri 06 Mar     , Hal Rosenstock wrote: >>>> >>>> It could be subsequent patch but thought it best to include it. Do you >>>> want this separate ? >>> >>> Without clear usage case I would prefer to not have it at all - finally >>> it is on a fast path. >> >> It's the same as what is done in the kernel for MAD response but I >> removed it because I could see you were against this. >> >>>> >>>> > BR is not used anywhere in OpenSM. >>>> >>>> No, but someone might use ib_types.h to build BM. >>> >>> We cannot know what will be needed then - this someone will need to care >>> anyway. >> >> You could say that about a lot of things accepted which aren't fully integrated. >> >>>> > And this function which should process TrapRepress method does nothing, >>>> > right? >>>> >>>> Just some validation; It doesn't need to do anything (just retire the >>>> transaction). >>> >>> Then I don't understand - how is trap 144 repress handled? And why >>> those changes in trap_rcv_process_response() were needed? >> >> Perhaps it's being overly pendantic. I can revise the patch to not >> have the repress get this far. > > I misread your comment. Maybe I didn't misread. > trap_rcv needs to cause the repress for the trap to be generated. This is already done. I think what you were saying was why does the incoming repress need to get this far into the code, right ? -- Hal > > -- Hal > >> -- Hal >> >>> Sasha >>> >> > From sashak at voltaire.com Fri Mar 6 13:19:08 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 6 Mar 2009 23:19:08 +0200 Subject: [ofa-general] RE: complib and cl_nodenamemap In-Reply-To: References: <49E9FBB3AEB64CEDA3577778C69E6392@amr.corp.intel.com> <20090306131701.GW6258@sashak.voltaire.com> <20090306105648.ff23413b.weiny2@llnl.gov> <4260917061B84A199895736482131117@amr.corp.intel.com> <20090306200702.GF9673@sashak.voltaire.com> Message-ID: <20090306211908.GH9673@sashak.voltaire.com> On 12:54 Fri 06 Mar , Sean Hefty wrote: > > If we do not want to move nodenamemap into a new library, then each package can > have its own copy. Eventually the ib-diags would not need to depend on opensm > being installed. I hate a code duplication, but in this specific case it looks as most practical solution for me. Sasha From sean.hefty at intel.com Fri Mar 6 13:30:20 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Fri, 6 Mar 2009 13:30:20 -0800 Subject: [ofa-general] RE: complib and cl_nodenamemap In-Reply-To: <20090306131219.7322d26a.weiny2@llnl.gov> References: <49E9FBB3AEB64CEDA3577778C69E6392@amr.corp.intel.com> <20090306131701.GW6258@sashak.voltaire.com> <20090306105648.ff23413b.weiny2@llnl.gov> <4260917061B84A199895736482131117@amr.corp.intel.com> <20090306200702.GF9673@sashak.voltaire.com> <20090306131219.7322d26a.weiny2@llnl.gov> Message-ID: >The idea was to make sure that any formatting changes to the config file would >be consistent between opensm and the diags. So far this has not happened. >However, IMHO I think it would be a step backward to have the code in 2 >places. I'm not sure we need to maintain duplicate code. I view that as a packaging issue. I.e. have a common directory that gets included by both packages. (I don't know how you'd actually do this, but I don't see why it can't work.) If we decide that, yeah, nodenamemap shouldn't be in complib, but there's not a reasonable alternative at this point, that's fine, but I at least want to see if there's something that can be done. - Sean From hnrose at comcast.net Fri Mar 6 12:50:55 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Fri, 6 Mar 2009 15:50:55 -0500 Subject: [ofa-general] [PATCH][TRIVIAL] ibsim: Some cosmetic changes Message-ID: <20090306205055.GA6261@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/ibsim/ibsim.c b/ibsim/ibsim.c index ea94fdf..e720f0d 100644 --- a/ibsim/ibsim.c +++ b/ibsim/ibsim.c @@ -110,11 +110,10 @@ static size_t make_name(union name_t *name, uint32_t addr, unsigned short port, static char *get_name(union name_t *name) { - if (remote_mode) { + if (remote_mode) return inet_ntoa(name->name_i.sin_addr); - } else { + else return name->name_u.sun_path + 1; - } } /** @@ -608,7 +607,7 @@ static int sim_run(int con_fd) socket_basename=getenv("IBSIM_SOCKNAME"); if(!socket_basename) - socket_basename=SIM_BASENAME; + socket_basename = SIM_BASENAME; if (sim_init_conn(socket_basename) < 0) return -1; diff --git a/ibsim/sim_cmd.c b/ibsim/sim_cmd.c index 94e0a14..bbd51b9 100644 --- a/ibsim/sim_cmd.c +++ b/ibsim/sim_cmd.c @@ -274,10 +274,9 @@ static void port_change_lid(Port * port, int lid, int lmc) if (lmc > 0) port->lmc = lmc; - if (port->node->type == SWITCH_NODE) { - if (port->node->sw) - port->node->sw->portchange = 1; - } else if (port->remotenode && port->remotenode->sw) + if (port->node->type == SWITCH_NODE && port->node->sw) + port->node->sw->portchange = 1; + else if (port->remotenode && port->remotenode->sw) port->remotenode->sw->portchange = 1; } @@ -580,9 +579,8 @@ static int dump_net(FILE * f, char *line) nports++; dump_switch(f, node->sw); } - for (i = 0; i < nports; i++) { + for (i = 0; i < nports; i++) dump_port(f, ports + node->portsbase + i, node->type); - } nnodes++; } diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c index 15ad8e0..d2c313c 100644 --- a/ibsim/sim_mad.c +++ b/ibsim/sim_mad.c @@ -222,7 +222,7 @@ static int do_cpi(Port * port, unsigned op, uint32_t mod, uint8_t * data) { int status = 0; - if (op != 1) // get + if (op != IB_MAD_METHOD_GET) status = ERR_METHOD_UNSUPPORTED; memset(data, 0, IB_SMP_DATA_SIZE); mad_set_field(data, 0, IB_CPI_BASEVER_F, 1); @@ -236,7 +236,7 @@ static int do_nodedesc(Port * port, unsigned op, uint32_t mod, uint8_t * data) { int status = 0; - if (op != 1) // get + if (op != IB_MAD_METHOD_GET) status = ERR_METHOD_UNSUPPORTED; memcpy(data, port->node->nodedesc, IB_SMP_DATA_SIZE); @@ -249,7 +249,7 @@ static int do_nodeinfo(Port * port, unsigned op, uint32_t mod, uint8_t * data) int status = 0; uint64_t portguid = node->nodeguid + port->portnum; - if (op != IB_MAD_METHOD_GET) // get + if (op != IB_MAD_METHOD_GET) status = ERR_METHOD_UNSUPPORTED; memcpy(data, node->nodeinfo, IB_SMP_DATA_SIZE); @@ -269,7 +269,7 @@ static int do_switchinfo(Port * port, unsigned op, uint32_t mod, uint8_t * data) if (!sw) // not a Switch? return ERR_ATTR_UNSUPPORTED; - if (op == 2) { // Set + if (op == IB_MAD_METHOD_SET) { if (mad_get_field(data, 0, IB_SW_STATE_CHANGE_F)) sw->portchange = 0; sw->linearFDBtop = @@ -335,11 +335,10 @@ static int do_sl2vl(Port * port, unsigned op, uint32_t mod, uint8_t * data) sl2vl = port->sl2vl + 8 * n; - if (op == IB_MAD_METHOD_SET) { + if (op == IB_MAD_METHOD_SET) memcpy(sl2vl, data, 8); - } else { + else memcpy(data, sl2vl, 8); - } return 0; } @@ -374,9 +373,9 @@ static int do_vlarb(Port * port, unsigned op, uint32_t mod, uint8_t * data) size *= sizeof(*vlarb); - if (op == IB_MAD_METHOD_SET) { + if (op == IB_MAD_METHOD_SET) memcpy(vlarb, data, size); - } else { + else { memset(data, 0, IB_SMP_DATA_SIZE); memcpy(data, vlarb, size); } @@ -421,7 +420,7 @@ do_portinfo(Port * port, unsigned op, uint32_t portnum, uint8_t * data) DEBUG("in node %" PRIx64 " port %" PRIx64 ": port %" PRIx64 " (%d(%d))", node->nodeguid, port->portguid, p->portguid, p->portnum, portnum); - if (op == IB_MAD_METHOD_SET) { // set + if (op == IB_MAD_METHOD_SET) { unsigned val; if (node->type != SWITCH_NODE && port->portnum != p->portnum) return ERR_BAD_PARAM; // on HCA or rtr can't "set" on other port @@ -497,10 +496,9 @@ static int do_linearforwtbl(Port * port, unsigned op, uint32_t mod, if (mod < 0 || mod > 767) return ERR_BAD_PARAM; - if (op == IB_MAD_METHOD_SET) { // Set + if (op == IB_MAD_METHOD_SET) mad_get_array(data, 0, IB_LINEAR_FORW_TBL_F, sw->fdb + mod * 64); - } mad_set_array(data, 0, IB_LINEAR_FORW_TBL_F, sw->fdb + mod * 64); @@ -527,10 +525,9 @@ static int do_multicastforwtbl(Port * port, unsigned op, uint32_t mod, } blockposition = (numBlock32 * NUMBEROFPORTMASK + numPortMsk) * 64; - if (op == IB_MAD_METHOD_SET) { // Set + if (op == IB_MAD_METHOD_SET) mad_get_array(data, 0, IB_MULTICAST_FORW_TBL_F, sw->mfdb + blockposition); - } mad_set_array(data, 0, IB_MULTICAST_FORW_TBL_F, sw->mfdb + blockposition); return 0; @@ -749,13 +746,10 @@ static int do_portcounters(Port * port, unsigned op, uint32_t unused, memset(&totals, 0, sizeof totals); for (i = 0; i <= node->numports; i++) { - if (!(p = node_get_port(node, i))) return ERR_BAD_PARAM; - if (op == IB_MAD_METHOD_SET) pc_reset(&p->portcounters, mask); - pc_sum(&totals, &p->portcounters); } @@ -1117,7 +1111,7 @@ static Port *route_MAD(Port * port, int response, int lid, ib_dr_path_t * path) direct_route_out_MAD(port, path); } -Smpfn *get_handle_fn(ib_rpc_t rpc, int response) +static Smpfn *get_handle_fn(ib_rpc_t rpc, int response) { Smpfn *fn; @@ -1127,14 +1121,14 @@ Smpfn *get_handle_fn(ib_rpc_t rpc, int response) if (rpc.mgtclass == IB_SMI_CLASS || rpc.mgtclass == IB_SMI_DIRECT_CLASS) { if (rpc.attr.id >= IB_ATTR_LAST || !(fn = attrs[rpc.mgtclass & 0xf][rpc.attr.id])) - return 0; // not supported attribute ??? + return 0; // attribute/method not supported ??? return fn; } if (rpc.mgtclass == IB_PERFORMANCE_CLASS) { if (rpc.attr.id >= IB_GSI_ATTR_LAST || !(fn = attrs[rpc.mgtclass & 0xf][rpc.attr.id])) - return 0; // not supported attribute ??? + return 0; // attribute/method not supported ??? return fn; } diff --git a/ibsim/sim_net.c b/ibsim/sim_net.c index aa91103..13c3b8c 100644 --- a/ibsim/sim_net.c +++ b/ibsim/sim_net.c @@ -405,7 +405,7 @@ static int is_linkwidth_valid(int width) static int is_linkspeed_valid(int speed) { - /* speed is 2.5G, 5.0G, or 10.0 G */ + /* speed is 2.5G, 5.0G, or 10.0G */ if (speed < 1 || speed > 7) { IBWARN("bad speed %d - should be between 1 to 7", speed); return 0; diff --git a/umad2sim/sim_client.c b/umad2sim/sim_client.c index 06bb7a8..eb42a7c 100644 --- a/umad2sim/sim_client.c +++ b/umad2sim/sim_client.c @@ -143,11 +143,10 @@ static size_t make_name(union name_t *name, char *host, unsigned port, static char *get_name(union name_t *name) { - if (remote_mode) { + if (remote_mode) return inet_ntoa(name->name_i.sin_addr); - } else { + else return name->name_u.sun_path + 1; - } } static int sim_attach(int fd, union name_t *name, size_t size) @@ -159,8 +158,7 @@ static int sim_attach(int fd, union name_t *name, size_t size) DEBUG("attempt to connect to %s (attempt %d)", get_name(name), retries); - if ((r = - connect(fd, (struct sockaddr *)name, size)) >= 0) + if ((r = connect(fd, (struct sockaddr *)name, size)) >= 0) break; if (r < 0 && errno == ECONNREFUSED) { @@ -217,7 +215,7 @@ static int sim_init(struct sim_client *sc, char *nodeid) socket_basename = getenv("IBSIM_SOCKNAME"); if(!socket_basename) - socket_basename=SIM_BASENAME; + socket_basename = SIM_BASENAME; if (connect_host && *connect_host) remote_mode = 1; diff --git a/umad2sim/umad2sim.c b/umad2sim/umad2sim.c index e13e30a..6b10f28 100644 --- a/umad2sim/umad2sim.c +++ b/umad2sim/umad2sim.c @@ -441,8 +441,8 @@ static ssize_t umad2sim_write(struct umad2sim_dev *dev, { static int err_count; if (++err_count == 15) return -1; - if (mad_get_field(umad_get_mad(umad), 0, IB_MAD_METHOD_F) == 0x7) { - printf("Drop trap repress...\n"); + if (mad_get_field(umad_get_mad(umad), 0, IB_MAD_METHOD_F) == IB_MAD_METHOD_TRAP_REPRESS) { + printf("Dropping trap repress...\n"); return -1; } } From rdreier at cisco.com Fri Mar 6 15:07:55 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Mar 2009 15:07:55 -0800 Subject: [ofa-general] Re: [PATCH] RDMA/nes: Make iw_nes non-LLTX driver In-Reply-To: <20090305204829.GA10488@flatif-MOBL> (Faisal Latif's message of "Thu, 5 Mar 2009 14:48:29 -0600") References: <20090305204829.GA10488@flatif-MOBL> Message-ID: Great to see LLTX taken out, applied. From rdreier at cisco.com Fri Mar 6 15:08:53 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Mar 2009 15:08:53 -0800 Subject: [ofa-general] Re: [PATCH 1/2] RDMA/nes: Register memory for LSMM In-Reply-To: <20090305205736.GA10436@flatif-MOBL> (Faisal Latif's message of "Thu, 5 Mar 2009 14:57:36 -0600") References: <20090305205736.GA10436@flatif-MOBL> Message-ID: > The LSMM is using STAG0. Modify the driver to register LSMM and > use the non-zero STAG in nes_accept(). Deregister it from nes_disconnect() as > it is called during connection termination. I can't make head or tails of this changelog. What is LSMM? Why do I care if it's using STAG0? Why do we want to change the driver to register LSMM? - R. From rdreier at cisco.com Fri Mar 6 15:13:43 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Mar 2009 15:13:43 -0800 Subject: [ofa-general] Re: [PATCH v2] RDMA/nes: Improved use of pbls In-Reply-To: <20090123212448.GA7968@ctung-MOBL> (Chien Tung's message of "Fri, 23 Jan 2009 15:24:48 -0600") References: <20090123212448.GA7968@ctung-MOBL> Message-ID: thanks, applied From rdreier at cisco.com Fri Mar 6 15:16:42 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Mar 2009 15:16:42 -0800 Subject: [ofa-general] Re: [PATCH 2/2] RDMA/nes: Handle MPA Reject message properly In-Reply-To: <20090305205936.GA10992@flatif-MOBL> (Faisal Latif's message of "Thu, 5 Mar 2009 14:59:36 -0600") References: <20090305205936.GA10992@flatif-MOBL> Message-ID: thanks, applied From rdreier at cisco.com Fri Mar 6 15:22:12 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Mar 2009 15:22:12 -0800 Subject: [ofa-general] [PATCH v2] RDMA/cxgb3: Handle EEH events for active connections. In-Reply-To: <20090223235445.21618.85001.stgit@build.ogc.int> (Steve Wise's message of "Mon, 23 Feb 2009 17:54:45 -0600") References: <20090223235445.21618.85001.stgit@build.ogc.int> Message-ID: > - post QP_FATAL event on all active QPs when cxgb3 notifies > iw_cxgb3 of a fatal error. This actually doesn't match what IB adapters do: if the whole adapter is being reset, mlx4 and mthca just report DEVICE_FATAL for the whole device. I think the QP_FATAL event should be when something gets screwed up so some work requests are lost (ie no completions will ever come) but the device as a whole is OK. As an example, the Open MPI code seems to handle DEVICE_FATAL but not do the same thing for QP_FATAL etc. - R. From rdreier at cisco.com Fri Mar 6 15:23:47 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Mar 2009 15:23:47 -0800 Subject: [ofa-general] Re: Too many calls to mlx4_CLOSE_PORT()? In-Reply-To: <20090223132008.GA1188@mtls03> (Eli Cohen's message of "Mon, 23 Feb 2009 15:20:08 +0200") References: <20090223132008.GA1188@mtls03> Message-ID: > browsing the code, I see that mlx4_CLOSE_PORT() gets called from, > seemingly, too many places. I would expect it to get called only from > __mlx4_ib_modify_qp() when QP0 gets closed, but mlx4_ib_remove() calls > it too even though it is soon to be called by __mlx4_ib_modify_qp() > due to destroying the MAD QP. It also gets called from > mlx4_remove_one() even though by the time this function gets called, > the port is already closed. Is there a reason for that? I don't think there's any special reason... maybe just paranoia that the MAD QP might not get destroyed in all cases. - R. From swise at opengridcomputing.com Fri Mar 6 15:46:39 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Fri, 06 Mar 2009 17:46:39 -0600 Subject: [ofa-general] [PATCH v2] RDMA/cxgb3: Handle EEH events for active connections. In-Reply-To: References: <20090223235445.21618.85001.stgit@build.ogc.int> Message-ID: <49B1B5DF.3030803@opengridcomputing.com> Roland Dreier wrote: > > - post QP_FATAL event on all active QPs when cxgb3 notifies > > iw_cxgb3 of a fatal error. > > This actually doesn't match what IB adapters do: if the whole adapter is > being reset, mlx4 and mthca just report DEVICE_FATAL for the whole > device. I think the QP_FATAL event should be when something gets > screwed up so some work requests are lost (ie no completions will ever > come) but the device as a whole is OK. > > As an example, the Open MPI code seems to handle DEVICE_FATAL but not do > the same thing for QP_FATAL etc. > > - R. > My last version of this patch posted DEVICE_FATAL and you said it wasn't right. I'm confused. From rdreier at cisco.com Fri Mar 6 17:05:58 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 06 Mar 2009 17:05:58 -0800 Subject: [ofa-general] [PATCH v2] RDMA/cxgb3: Handle EEH events for active connections. In-Reply-To: <49B1B5DF.3030803@opengridcomputing.com> (Steve Wise's message of "Fri, 06 Mar 2009 17:46:39 -0600") References: <20090223235445.21618.85001.stgit@build.ogc.int> <49B1B5DF.3030803@opengridcomputing.com> Message-ID: > My last version of this patch posted DEVICE_FATAL and you said it > wasn't right. DEVICE_FATAL is right but it should be an unaffiliated event... ie rather than calling the QP event handler for every QP, just call ib_dispatch_event() with one DEVICE_FATAL event. See eg drivers/infiniband/hw/mlx4/main.c in mlx4_ib_event(). What I said last time was: | This doesn't match the IB driver behavior (or the IB spec) -- the | DEVICE_FATAL event is unaffiliated and delivered for the adapter as a | whole. QP events are supposed to be for events connected to a single | QP, not the whole adapter failing. and I stand by that... QP events are for things connected to one QP, but a PCI EEH event is not affiliated with any individual object but rather the adapter as a whole. I think you misunderstood my point -- I wasn't saying to make it a QP event, but rather to make it unaffiliated. - R. From swise at opengridcomputing.com Fri Mar 6 17:32:59 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Fri, 06 Mar 2009 19:32:59 -0600 Subject: [ofa-general] [PATCH v2] RDMA/cxgb3: Handle EEH events for active connections. In-Reply-To: References: <20090223235445.21618.85001.stgit@build.ogc.int> <49B1B5DF.3030803@opengridcomputing.com> Message-ID: <49B1CECB.7010901@opengridcomputing.com> Roland Dreier wrote: > > My last version of this patch posted DEVICE_FATAL and you said it > > wasn't right. > > DEVICE_FATAL is right but it should be an unaffiliated event... ie > rather than calling the QP event handler for every QP, just call > ib_dispatch_event() with one DEVICE_FATAL event. See eg > drivers/infiniband/hw/mlx4/main.c in mlx4_ib_event(). What I said last > time was: > > | This doesn't match the IB driver behavior (or the IB spec) -- the > | DEVICE_FATAL event is unaffiliated and delivered for the adapter as a > | whole. QP events are supposed to be for events connected to a single > | QP, not the whole adapter failing. > > and I stand by that... QP events are for things connected to one QP, but > a PCI EEH event is not affiliated with any individual object but rather > the adapter as a whole. I think you misunderstood my point -- I wasn't > saying to make it a QP event, but rather to make it unaffiliated. > > - R. > I see. Thanks for the clarification. I didn't know about ib_dispatch_event(). I'll rework this asap. I'd like this in 2.6.30. Steve. From phillipwils at gmail.com Fri Mar 6 20:16:09 2009 From: phillipwils at gmail.com (Phillip Wilson) Date: Fri, 6 Mar 2009 20:16:09 -0800 Subject: [ofa-general] ***SPAM*** mlx4_core 0000:c3:00.0: SW2HW_MPT failed (-16) (dmesg) Message-ID: <6e4f44220903062016q8ca6d44ha22b10231ead5b3d@mail.gmail.com> This is related to the thread I stared on Feb 24 Re: [ofa-general] ***SPAM*** Mellanox ibv_reg_mr (memory region) function call fails under load when using the mlx4 driver So far I have modified the "num_mtt" to 1 << 21, as Dotan suggested, but I will try some more values this weekend. I think the code for this is in ../drivers/net/mlx4/cmd.c mlx4_cmd( ... ) -ETIMEOUT > System Information: > > > > The system has 4GB of memory. > > > > uname -a > > Linux (none) 2.6.24.02.02.08 #21 SMP Thu Feb 19 11:04:35 PST 2009 ia64 > unknown > > > > OFED 1.2.5 > > > > lspci -d 15b3: > > > > 0000:10:00.0 InfiniBand: Mellanox Technologies MT25208 InfiniHost III Ex > (Tavor compatibility mode) (rev 20) > > 0000:c3:00.0 InfiniBand: Mellanox Technologies: Unknown device 634a (rev a0) > > > > lspci -d 15b3: -n > > 0000:10:00.0 0c06: 15b3:6278 (rev 20) > > 0000:c3:00.0 0c06: 15b3:634a (rev a0) > > > > ibv_devinfo -v > > hca_id: mlx4_0 > > fw_ver: 2.5.000 > > > > hca_id: mthca0 > > fw_ver: 4.8.930 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dotanba at gmail.com Sat Mar 7 00:55:28 2009 From: dotanba at gmail.com (Dotan Barak) Date: Sat, 07 Mar 2009 10:55:28 +0200 Subject: [ofa-general] ***SPAM*** Re: mlx4_core 0000:c3:00.0: SW2HW_MPT failed (-16) (dmesg) In-Reply-To: <6e4f44220903062016q8ca6d44ha22b10231ead5b3d@mail.gmail.com> References: <6e4f44220903062016q8ca6d44ha22b10231ead5b3d@mail.gmail.com> Message-ID: <49B23680.1030508@gmail.com> Please make sure that you have the latest firmware for the HCA that has the failure. Dotan Phillip Wilson wrote: > This is related to the thread I stared on Feb 24 > > Re: [ofa-general] ***SPAM*** Mellanox ibv_reg_mr (memory region) > function call fails under load when using the mlx4 driver > > So far I have modified the "num_mtt" to 1 << 21, as Dotan suggested, > but I will try some more values this weekend. > > I think the code for this is in ../drivers/net/mlx4/cmd.c > mlx4_cmd( ... ) > > > -ETIMEOUT > > > > System Information: > > > > > > > > The system has 4GB of memory. > > > > > > > > uname -a > > > > Linux (none) 2.6.24.02.02.08 #21 SMP Thu Feb 19 11:04:35 PST 2009 ia64 > > unknown > > > > > > > > OFED 1.2.5 > > > > > > > > lspci -d 15b3: > > > > > > > > 0000:10:00.0 InfiniBand: Mellanox Technologies MT25208 InfiniHost III Ex > > (Tavor compatibility mode) (rev 20) > > > > 0000:c3:00.0 InfiniBand: Mellanox Technologies: Unknown device 634a > (rev a0) > > > > > > > > lspci -d 15b3: -n > > > > 0000:10:00.0 0c06: 15b3:6278 (rev 20) > > > > 0000:c3:00.0 0c06: 15b3:634a (rev a0) > > > > > > > > ibv_devinfo -v > > > > hca_id: mlx4_0 > > > > fw_ver: 2.5.000 > > > > > > > > hca_id: mthca0 > > > > fw_ver: 4.8.930 From vlad at lists.openfabrics.org Sat Mar 7 03:17:49 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sat, 7 Mar 2009 03:17:49 -0800 (PST) Subject: [ofa-general] ofa_1_4_kernel 20090307-0200 daily build status Message-ID: <20090307111749.D21A5E60FB5@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From sashak at voltaire.com Sat Mar 7 09:47:32 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 7 Mar 2009 19:47:32 +0200 Subject: [ofa-general] Re: [PATCH 3/10] infiniband-diags: convert ibping to "new" ibmad interface In-Reply-To: <20090219190532.faf400f5.weiny2@llnl.gov> References: <20090219190532.faf400f5.weiny2@llnl.gov> Message-ID: <20090307174724.GI9673@sashak.voltaire.com> On 19:05 Thu 19 Feb , Ira Weiny wrote: > diff --git a/libibmad/src/vendor.c b/libibmad/src/vendor.c > index 50a878e..1a129e5 100644 > --- a/libibmad/src/vendor.c > +++ b/libibmad/src/vendor.c > @@ -40,6 +40,7 @@ > #include > > #include > +#include "mad_internal.h" > > #undef DEBUG > #define DEBUG if (ibdebug) IBWARN > @@ -53,6 +54,16 @@ static inline int response_expected(int method) > uint8_t *ib_vendor_call(void *data, ib_portid_t * portid, > ib_vendor_call_t * call) > { > + struct ibmad_port port; > + > + port.port_id = madrpc_portid(); This breaks 'vendstat' since class_agents array is not initialized. So I'm adding this: + port.class_agents[call->mgmt_class] = mad_class_agent(call->mgmt_class); Sasha > + return ib_vendor_call_via(data, portid, call, &port); > +} > + > +uint8_t *ib_vendor_call_via(void *data, ib_portid_t * portid, > + ib_vendor_call_t * call, > + struct ibmad_port *srcport) > +{ > ib_rpc_t rpc = { 0 }; > int range1 = 0, resp_expected; > From sashak at voltaire.com Sat Mar 7 11:16:47 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 7 Mar 2009 21:16:47 +0200 Subject: [ofa-general] [PATCH 11/10] libibmad:infiniband-diags: deprecate madrpc_set_[retries|timeout] WAS: [PATCH 1/10] libibmad: Clean up "new" interface In-Reply-To: <20090302111101.bc822db8.weiny2@llnl.gov> References: <20090219190525.322681b8.weiny2@llnl.gov> <20090220143402.c3b23b0a.weiny2@llnl.gov> <20090301072622.GG3936@sashak.voltaire.com> <20090302111101.bc822db8.weiny2@llnl.gov> Message-ID: <20090307191647.GJ9673@sashak.voltaire.com> On 11:11 Mon 02 Mar , Ira Weiny wrote: > > I see this as well. Right now libibmad is designed around the "run and exit" > diag model but we are moving it toward a library which can be used in more > complex applications. We should push to do this once and as correct as > possible. We can support both models, but we shouldn't complicate existing one, so enforcing all existing utils to call explicitly mad_rpc_set_timeout() is not a good idea IMO. BTW, this patch now breaks most of infiniband-diags tools when timeout is not specified in command line explicitly with '-t'. The default used is ibd_timeout, which is '0' and umad_send() fails with 'Resource temporarily unavailable'. Sasha From sssmolniy at mail.ru Sat Mar 7 11:22:35 2009 From: sssmolniy at mail.ru (Sergey) Date: Sat, 07 Mar 2009 22:22:35 +0300 Subject: [ofa-general] ***SPAM*** mvapich 1.1 and multipath Message-ID: Hi, Sorry to my english May be anybody know about this ? How mpi process addressing the message to destination (rank 0 send rank 1)? Through LID, GID, QP ID or anything else? Where from mpi sender to know destination LID ? Can I strictly define outgoing LIDs for specific mpi process ? For example, with LMC = 3, a sender: rank 0, lid 1 communicate with rank 1 and rank 2 I want strictly define using multilid 4 for sending rank 0 to rank 1 (multilid 4 = DestLID for rank0 -> rank1) Thanks for all With best regards From sashak at voltaire.com Sat Mar 7 11:27:44 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 7 Mar 2009 21:27:44 +0200 Subject: [ofa-general] Re: [PATCH 0/10 libibmad/infiniband-diags -- converting to "new" interface. In-Reply-To: <20090219190520.c18280e1.weiny2@llnl.gov> References: <20090219190520.c18280e1.weiny2@llnl.gov> Message-ID: <20090307192744.GK9673@sashak.voltaire.com> On 19:05 Thu 19 Feb , Ira Weiny wrote: > Here is v2 of the patch series. Finally I applied patches 1-10 and 12 (with rebasing and described changes). Thanks. Sasha From sashak at voltaire.com Sat Mar 7 13:48:15 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 7 Mar 2009 23:48:15 +0200 Subject: [ofa-general] [PATCH] libibmad: cleanup deprecated function use In-Reply-To: <20090306080624.GD6258@sashak.voltaire.com> References: <20090219190525.322681b8.weiny2@llnl.gov> <20090301070013.GF3936@sashak.voltaire.com> <20090302095459.7e6d4a34.weiny2@llnl.gov> <20090306080624.GD6258@sashak.voltaire.com> Message-ID: <20090307214815.GL9673@sashak.voltaire.com> This cleans up libibmad internal stuff from using potentially deprecated functions, which may go away soon. Also removes DEPRECATED attribute from mad_register_port_client() - it doesn't touch any global data. Signed-off-by: Sasha Khapyorsky --- libibmad/include/infiniband/mad.h | 3 +- libibmad/src/gs.c | 12 +----- libibmad/src/libibmad.map | 1 - libibmad/src/mad_internal.h | 2 + libibmad/src/register.c | 68 ++++--------------------------------- libibmad/src/resolve.c | 7 ++-- libibmad/src/rpc.c | 41 +++++++-------------- libibmad/src/sa.c | 15 +++++--- libibmad/src/serv.c | 19 ++--------- libibmad/src/smp.c | 17 +++------ libibmad/src/vendor.c | 6 +--- 11 files changed, 48 insertions(+), 143 deletions(-) diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h index ed2412c..064cbb7 100644 --- a/libibmad/include/infiniband/mad.h +++ b/libibmad/include/infiniband/mad.h @@ -745,7 +745,7 @@ MAD_EXPORT int mad_rpc_portid(struct ibmad_port *srcport); /* register.c */ MAD_EXPORT int mad_register_port_client(int port_id, int mgmt, - uint8_t rmpp_version) DEPRECATED; + uint8_t rmpp_version); MAD_EXPORT int mad_register_client(int mgmt, uint8_t rmpp_version) DEPRECATED; MAD_EXPORT int mad_register_server(int mgmt, uint8_t rmpp_version, @@ -759,7 +759,6 @@ MAD_EXPORT int mad_register_server_via(int mgmt, uint8_t rmpp_version, uint32_t class_oui, struct ibmad_port *srcport); MAD_EXPORT int mad_class_agent(int mgmt); -MAD_EXPORT int mad_agent_class(int agent); /* serv.c */ MAD_EXPORT int mad_send(ib_rpc_t * rpc, ib_portid_t * dport, diff --git a/libibmad/src/gs.c b/libibmad/src/gs.c index 132187a..dbca9e9 100644 --- a/libibmad/src/gs.c +++ b/libibmad/src/gs.c @@ -74,11 +74,7 @@ uint8_t *pma_query_via(void *rcvbuf, ib_portid_t * dest, int port, if (!dest->qkey) dest->qkey = IB_DEFAULT_QP1_QKEY; - if (srcport) { - return mad_rpc(srcport, &rpc, dest, rcvbuf, rcvbuf); - } else { - return madrpc(&rpc, dest, rcvbuf, rcvbuf); - } + return mad_rpc(srcport, &rpc, dest, rcvbuf, rcvbuf); } uint8_t *performance_reset_via(void *rcvbuf, ib_portid_t * dest, @@ -117,9 +113,5 @@ uint8_t *performance_reset_via(void *rcvbuf, ib_portid_t * dest, if (!dest->qkey) dest->qkey = IB_DEFAULT_QP1_QKEY; - if (srcport) { - return mad_rpc(srcport, &rpc, dest, rcvbuf, rcvbuf); - } else { - return madrpc(&rpc, dest, rcvbuf, rcvbuf); - } + return mad_rpc(srcport, &rpc, dest, rcvbuf, rcvbuf); } diff --git a/libibmad/src/libibmad.map b/libibmad/src/libibmad.map index 70f6f0f..0be7a92 100644 --- a/libibmad/src/libibmad.map +++ b/libibmad/src/libibmad.map @@ -53,7 +53,6 @@ IBMAD_1.3 { portid2str; str2drpath; drpath2str; - mad_agent_class; mad_class_agent; mad_register_client; mad_register_server; diff --git a/libibmad/src/mad_internal.h b/libibmad/src/mad_internal.h index 9afe7a9..e948540 100644 --- a/libibmad/src/mad_internal.h +++ b/libibmad/src/mad_internal.h @@ -41,4 +41,6 @@ struct ibmad_port { int class_agents[MAX_CLASS]; /* class2agent mapper */ }; +extern struct ibmad_port *ibmp; + #endif /* _MAD_INTERNAL_H_ */ diff --git a/libibmad/src/register.c b/libibmad/src/register.c index 4aabd7c..eb8a62e 100644 --- a/libibmad/src/register.c +++ b/libibmad/src/register.c @@ -48,33 +48,6 @@ #undef DEBUG #define DEBUG if (ibdebug) IBWARN -#define MAX_AGENTS 256 - -static int class_agent[MAX_CLASS]; -static int agent_class[MAX_AGENTS]; - -static int register_agent(int agent, int mclass) -{ - static int initialized; - - if (!initialized) { - initialized++; - memset(class_agent, 0xff, sizeof class_agent); - memset(agent_class, 0xff, sizeof agent_class); - } - - if (mclass < 0 || mclass >= MAX_CLASS || - agent < 0 || agent >= MAX_AGENTS) { - DEBUG("bad mgmt class %d or agent %d", mclass, agent); - return -1; - } - - class_agent[mclass] = agent; - agent_class[agent] = mclass; - - return 0; -} - static int mgmt_class_vers(int mgmt_class) { if ((mgmt_class >= IB_VENDOR_RANGE1_START_CLASS && @@ -104,14 +77,7 @@ int mad_class_agent(int mgmt) { if (mgmt < 1 || mgmt > MAX_CLASS) return -1; - return class_agent[mgmt]; -} - -int mad_agent_class(int agent) -{ - if (agent < 1 || agent > MAX_AGENTS) - return -1; - return agent_class[agent]; + return ibmp->class_agents[mgmt]; } int mad_register_port_client(int port_id, int mgmt, uint8_t rmpp_version) @@ -122,14 +88,10 @@ int mad_register_port_client(int port_id, int mgmt, uint8_t rmpp_version) DEBUG("Unknown class %d mgmt_class", mgmt); return -1; } - if ((agent = umad_register(port_id, mgmt, vers, rmpp_version, 0)) < 0) { - DEBUG("Can't register agent for class %d", mgmt); - return -1; - } - if (mgmt < 0 || mgmt >= MAX_CLASS || agent >= MAX_AGENTS) { - DEBUG("bad mgmt class %d or agent %d", mgmt, agent); - return -1; + agent = umad_register(port_id, mgmt, vers, rmpp_version, 0); + if (agent < 0) { + DEBUG("Can't register agent for class %d", mgmt); } return agent; @@ -137,14 +99,7 @@ int mad_register_port_client(int port_id, int mgmt, uint8_t rmpp_version) int mad_register_client(int mgmt, uint8_t rmpp_version) { - int rc = 0; - struct ibmad_port port; - - port.port_id = madrpc_portid(); - rc = mad_register_client_via(mgmt, rmpp_version, &port); - if (rc < 0) - return rc; - return register_agent(port.class_agents[mgmt], mgmt); + return mad_register_client_via(mgmt, rmpp_version, ibmp); } int mad_register_client_via(int mgmt, uint8_t rmpp_version, @@ -167,17 +122,8 @@ int mad_register_server(int mgmt, uint8_t rmpp_version, long method_mask[], uint32_t class_oui) { - int rc = 0; - struct ibmad_port port; - - port.port_id = madrpc_portid(); - port.class_agents[mgmt] = class_agent[mgmt]; - rc = mad_register_server_via(mgmt, rmpp_version, - method_mask, class_oui, - &port); - if (rc < 0) - return rc; - return register_agent(port.class_agents[mgmt], mgmt); + return mad_register_server_via(mgmt, rmpp_version, method_mask, + class_oui, ibmp); } int diff --git a/libibmad/src/resolve.c b/libibmad/src/resolve.c index 3291f43..f34c247 100644 --- a/libibmad/src/resolve.c +++ b/libibmad/src/resolve.c @@ -41,6 +41,7 @@ #include #include +#include "mad_internal.h" #undef DEBUG #define DEBUG if (ibdebug) IBWARN @@ -64,7 +65,7 @@ int ib_resolve_smlid_via(ib_portid_t * sm_id, int timeout, int ib_resolve_smlid(ib_portid_t * sm_id, int timeout) { - return ib_resolve_smlid_via(sm_id, timeout, NULL); + return ib_resolve_smlid_via(sm_id, timeout, ibmp); } int ib_resolve_guid_via(ib_portid_t * portid, uint64_t * guid, @@ -148,7 +149,7 @@ int ib_resolve_portid_str(ib_portid_t * portid, char *addr_str, enum MAD_DEST dest_type, ib_portid_t * sm_id) { return ib_resolve_portid_str_via(portid, addr_str, dest_type, - sm_id, NULL); + sm_id, ibmp); } int ib_resolve_self_via(ib_portid_t * portid, int *portnum, ibmad_gid_t * gid, @@ -180,5 +181,5 @@ int ib_resolve_self_via(ib_portid_t * portid, int *portnum, ibmad_gid_t * gid, int ib_resolve_self(ib_portid_t * portid, int *portnum, ibmad_gid_t * gid) { - return ib_resolve_self_via(portid, portnum, gid, NULL); + return ib_resolve_self_via(portid, portnum, gid, ibmp); } diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c index 210f0c2..8c68cf9 100644 --- a/libibmad/src/rpc.c +++ b/libibmad/src/rpc.c @@ -47,7 +47,9 @@ int ibdebug; -static int mad_portid = -1; +static struct ibmad_port mad_port; +struct ibmad_port *ibmp = &mad_port; + static int iberrs; static int madrpc_retries = MAD_DEF_RETRIES; @@ -92,7 +94,7 @@ int madrpc_def_timeout(void) int madrpc_portid(void) { - return mad_portid; + return ibmp->port_id; } int mad_rpc_portid(struct ibmad_port *srcport) @@ -260,42 +262,38 @@ void *mad_rpc_rmpp(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * void *madrpc(ib_rpc_t * rpc, ib_portid_t * dport, void *payload, void *rcvdata) { - struct ibmad_port port; - - port.port_id = mad_portid; - port.class_agents[rpc->mgtclass] = mad_class_agent(rpc->mgtclass); - return mad_rpc(&port, rpc, dport, payload, rcvdata); + return mad_rpc(ibmp, rpc, dport, payload, rcvdata); } void *madrpc_rmpp(ib_rpc_t * rpc, ib_portid_t * dport, ib_rmpp_hdr_t * rmpp, void *data) { - struct ibmad_port port; - - port.port_id = mad_portid; - port.class_agents[rpc->mgtclass] = mad_class_agent(rpc->mgtclass); - return mad_rpc_rmpp(&port, rpc, dport, rmpp, data); + return mad_rpc_rmpp(ibmp, rpc, dport, rmpp, data); } void madrpc_init(char *dev_name, int dev_port, int *mgmt_classes, int num_classes) { + int fd; + if (umad_init() < 0) IBPANIC("can't init UMAD library"); - if ((mad_portid = umad_open_port(dev_name, dev_port)) < 0) + if ((fd = umad_open_port(dev_name, dev_port)) < 0) IBPANIC("can't open UMAD port (%s:%d)", dev_name, dev_port); if (num_classes >= MAX_CLASS) IBPANIC("too many classes %d requested", num_classes); + ibmp->port_id = fd; + memset(ibmp->class_agents, 0xff, sizeof ibmp->class_agents); while (num_classes--) { uint8_t rmpp_version = 0; int mgmt = *mgmt_classes++; if (mgmt == IB_SA_CLASS) rmpp_version = 1; - if (mad_register_client(mgmt, rmpp_version) < 0) + if (mad_register_client_via(mgmt, rmpp_version, ibmp) < 0) IBPANIC("client_register for mgmt class %d failed", mgmt); } @@ -334,6 +332,7 @@ struct ibmad_port *mad_rpc_open_port(char *dev_name, int dev_port, return NULL; } + p->port_id = port_id; memset(p->class_agents, 0xff, sizeof p->class_agents); while (num_classes--) { uint8_t rmpp_version = 0; @@ -343,8 +342,7 @@ struct ibmad_port *mad_rpc_open_port(char *dev_name, int dev_port, if (mgmt == IB_SA_CLASS) rmpp_version = 1; if (mgmt < 0 || mgmt >= MAX_CLASS || - (agent = mad_register_port_client(port_id, mgmt, - rmpp_version)) < 0) { + (agent = mad_register_client_via(mgmt, rmpp_version, p)) < 0) { IBWARN("client_register for mgmt %d failed", mgmt); if (!errno) errno = EINVAL; @@ -355,7 +353,6 @@ struct ibmad_port *mad_rpc_open_port(char *dev_name, int dev_port, p->class_agents[mgmt] = agent; } - p->port_id = port_id; return p; } @@ -364,13 +361,3 @@ void mad_rpc_close_port(struct ibmad_port *port) umad_close_port(port->port_id); free(port); } - -uint8_t *sa_call(void *rcvbuf, ib_portid_t * portid, ib_sa_call_t * sa, - unsigned timeout) -{ - struct ibmad_port port; - - port.port_id = mad_portid; - port.class_agents[IB_SA_CLASS] = mad_class_agent(IB_SA_CLASS); - return sa_rpc_call(&port, rcvbuf, portid, sa, timeout); -} diff --git a/libibmad/src/sa.c b/libibmad/src/sa.c index ddeb152..5174bb5 100644 --- a/libibmad/src/sa.c +++ b/libibmad/src/sa.c @@ -40,6 +40,7 @@ #include #include +#include "mad_internal.h" #undef DEBUG #define DEBUG if (ibdebug) IBWARN @@ -79,6 +80,12 @@ uint8_t *sa_rpc_call(const struct ibmad_port *ibmad_port, void *rcvbuf, ib_porti return p; } +uint8_t *sa_call(void *rcvbuf, ib_portid_t * portid, ib_sa_call_t * sa, + unsigned timeout) +{ + return sa_rpc_call(ibmp, rcvbuf, portid, sa, timeout); +} + /* PathRecord */ #define IB_PR_COMPMASK_DGID (1ull<<2) #define IB_PR_COMPMASK_SGID (1ull<<3) @@ -127,11 +134,7 @@ int ib_path_query_via(const struct ibmad_port *srcport, ibmad_gid_t srcgid, mad_encode_field(buf, IB_SA_PR_DGID_F, destgid); mad_encode_field(buf, IB_SA_PR_SGID_F, srcgid); - if (srcport) { - p = sa_rpc_call(srcport, buf, sm_id, &sa, 0); - } else { - p = sa_call(buf, sm_id, &sa, 0); - } + p = sa_rpc_call(srcport, buf, sm_id, &sa, 0); if (!p) { IBWARN("sa call path_query failed"); return -1; @@ -144,5 +147,5 @@ int ib_path_query_via(const struct ibmad_port *srcport, ibmad_gid_t srcgid, int ib_path_query(ibmad_gid_t srcgid, ibmad_gid_t destgid, ib_portid_t * sm_id, void *buf) { - return ib_path_query_via(NULL, srcgid, destgid, sm_id, buf); + return ib_path_query_via(ibmp, srcgid, destgid, sm_id, buf); } diff --git a/libibmad/src/serv.c b/libibmad/src/serv.c index 0ce1660..c97466a 100644 --- a/libibmad/src/serv.c +++ b/libibmad/src/serv.c @@ -50,11 +50,7 @@ int mad_send(ib_rpc_t * rpc, ib_portid_t * dport, ib_rmpp_hdr_t * rmpp, void *data) { - struct ibmad_port port; - - port.port_id = madrpc_portid(); - port.class_agents[rpc->mgtclass] = mad_class_agent(rpc->mgtclass); - return mad_send_via(rpc, dport, rmpp, data, &port); + return mad_send_via(rpc, dport, rmpp, data, ibmp); } int @@ -88,13 +84,7 @@ mad_send_via(ib_rpc_t * rpc, ib_portid_t * dport, ib_rmpp_hdr_t * rmpp, void *da int mad_respond(void *umad, ib_portid_t * portid, uint32_t rstatus) { - int i = 0; - struct ibmad_port port; - - port.port_id = madrpc_portid(); - for (i = 1; i < MAX_CLASS; i++) - port.class_agents[i] = mad_class_agent(i); - return mad_respond_via(umad, portid, rstatus, &port); + return mad_respond_via(umad, portid, rstatus, ibmp); } int mad_respond_via(void *umad, ib_portid_t * portid, uint32_t rstatus, @@ -174,10 +164,7 @@ int mad_respond_via(void *umad, ib_portid_t * portid, uint32_t rstatus, void *mad_receive(void *umad, int timeout) { - struct ibmad_port port; - - port.port_id = madrpc_portid(); - return mad_receive_via(umad, timeout, &port); + return mad_receive_via(umad, timeout, ibmp); } void *mad_receive_via(void *umad, int timeout, struct ibmad_port *srcport) diff --git a/libibmad/src/smp.c b/libibmad/src/smp.c index e5489b3..4983786 100644 --- a/libibmad/src/smp.c +++ b/libibmad/src/smp.c @@ -40,6 +40,7 @@ #include #include +#include "mad_internal.h" #undef DEBUG #define DEBUG if (ibdebug) IBWARN @@ -67,17 +68,13 @@ uint8_t *smp_set_via(void *data, ib_portid_t * portid, unsigned attrid, portid->sl = 0; portid->qp = 0; - if (srcport) { - return mad_rpc(srcport, &rpc, portid, data, data); - } else { - return madrpc(&rpc, portid, data, data); - } + return mad_rpc(srcport, &rpc, portid, data, data); } uint8_t *smp_set(void *data, ib_portid_t * portid, unsigned attrid, unsigned mod, unsigned timeout) { - return smp_set_via(data, portid, attrid, mod, timeout, NULL); + return smp_set_via(data, portid, attrid, mod, timeout, ibmp); } uint8_t *smp_query_via(void *rcvbuf, ib_portid_t * portid, unsigned attrid, @@ -103,15 +100,11 @@ uint8_t *smp_query_via(void *rcvbuf, ib_portid_t * portid, unsigned attrid, portid->sl = 0; portid->qp = 0; - if (srcport) { - return mad_rpc(srcport, &rpc, portid, 0, rcvbuf); - } else { - return madrpc(&rpc, portid, 0, rcvbuf); - } + return mad_rpc(srcport, &rpc, portid, 0, rcvbuf); } uint8_t *smp_query(void *rcvbuf, ib_portid_t * portid, unsigned attrid, unsigned mod, unsigned timeout) { - return smp_query_via(rcvbuf, portid, attrid, mod, timeout, NULL); + return smp_query_via(rcvbuf, portid, attrid, mod, timeout, ibmp); } diff --git a/libibmad/src/vendor.c b/libibmad/src/vendor.c index 2610976..31726fc 100644 --- a/libibmad/src/vendor.c +++ b/libibmad/src/vendor.c @@ -54,11 +54,7 @@ static inline int response_expected(int method) uint8_t *ib_vendor_call(void *data, ib_portid_t * portid, ib_vendor_call_t * call) { - struct ibmad_port port; - - port.port_id = madrpc_portid(); - port.class_agents[call->mgmt_class] = mad_class_agent(call->mgmt_class); - return ib_vendor_call_via(data, portid, call, &port); + return ib_vendor_call_via(data, portid, call, ibmp); } uint8_t *ib_vendor_call_via(void *data, ib_portid_t * portid, -- 1.6.1.2.319.gbd9e From sashak at voltaire.com Sat Mar 7 13:51:58 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 7 Mar 2009 23:51:58 +0200 Subject: [ofa-general] Re: [PATCH 0/10 libibmad/infiniband-diags -- converting to "new" interface. In-Reply-To: <20090307192744.GK9673@sashak.voltaire.com> References: <20090219190520.c18280e1.weiny2@llnl.gov> <20090307192744.GK9673@sashak.voltaire.com> Message-ID: <20090307215151.GM9673@sashak.voltaire.com> On 21:27 Sat 07 Mar , Sasha Khapyorsky wrote: > On 19:05 Thu 19 Feb , Ira Weiny wrote: > > Here is v2 of the patch series. > > Finally I applied patches 1-10 and 12 (with rebasing and described > changes). Thanks. In addition to all it seems that mad_class_agent() function should be deprecated (it uses internal static data) and its usage converted. Sasha From koop at cse.ohio-state.edu Sat Mar 7 21:11:31 2009 From: koop at cse.ohio-state.edu (Matthew Koop) Date: Sun, 8 Mar 2009 00:11:31 -0500 (EST) Subject: [ofa-general] ***SPAM*** mvapich 1.1 and multipath In-Reply-To: Message-ID: Sergey, We have answered to your similar questions on mvapich-discuss list. Below is a copy of the reply. Matt *** Messages are sent based on LIDs and QPs. Right now there is no mechanism to set output LIDs directly. If you use the VIADEV_USE_LMC=1 (MVAPICH) or MV2_USE_HSAM=1 (MVAPICH2) it should try to spread traffic for you to maximize performance. What is the reason for needing this fine-grained control? Thanks, Matt On Sat, 7 Mar 2009, Sergey wrote: > Hi, > Sorry to my english > May be anybody know about this ? > How mpi process addressing the message to destination (rank 0 send rank 1)? Through LID, GID, QP ID or anything else? Where from mpi sender to know destination LID ? > Can I strictly define outgoing LIDs for specific mpi process ? > For example, with LMC = 3, a sender: rank 0, lid 1 communicate with rank 1 and rank 2 > I want strictly define using multilid 4 for sending rank 0 to rank 1 (multilid 4 = DestLID for rank0 -> rank1) > Thanks for all > With best regards > > > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From phillipwils at gmail.com Sun Mar 8 00:14:48 2009 From: phillipwils at gmail.com (Phillip Wilson) Date: Sun, 8 Mar 2009 00:14:48 -0800 Subject: [ofa-general] ***SPAM*** Re: mlx4_core 0000:c3:00.0: SW2HW_MPT failed (-16) (dmesg) In-Reply-To: <49B23680.1030508@gmail.com> References: <6e4f44220903062016q8ca6d44ha22b10231ead5b3d@mail.gmail.com> <49B23680.1030508@gmail.com> Message-ID: <6e4f44220903080014x49591fc9idc21000b107f8771@mail.gmail.com> I updated the HCA "InfiniBand: Mellanox Technologies: Unknown device 634a (rev a0)" to the latest firmware and issue remains. "fw_ver" is now 2.6.000. Any ideas on why the time out is occuring in the function? 203static int mlx4_cmd_poll(struct mlx4_dev *dev, u64 in_param, u64 *out_param, 204 int out_is_imm, u32 in_modifier, u8 op_modifier, 205 u16 op, unsigned long timeout) 206{ 207 struct mlx4_priv *priv = mlx4_priv(dev); 208 void __iomem *hcr = priv->cmd.hcr; 209 int err = 0; 210 unsigned long end; 211 212 down(&priv->cmd.poll_sem); 213 214 err = mlx4_cmd_post(dev, in_param, out_param ? *out_param : 0, 215 in_modifier, op_modifier, op, CMD_POLL_TOKEN, 0); 216 if (err) 217 goto out; 218 219 end = msecs_to_jiffies(timeout) + jiffies; 220 while (cmd_pending(dev) && time_before(jiffies, end)) 221 cond_resched(); 222 223 if (cmd_pending(dev)) { 224 err = - ETIMEDOUT; 225 goto out; 226 } 227 228 if (out_is_imm) 229 *out_param = 230 (u64) be32_to_cpu((__force __be32) 231 __raw_readl(hcr + HCR_OUT_PARAM_OFFSET)) << 32 | 232 (u64) be32_to_cpu((__force __be32) 233 __raw_readl(hcr + HCR_OUT_PARAM_OFFSET + 4)); 234 235 err = mlx4_status_to_errno(be32_to_cpu((__force __be32) 236 __raw_readl(hcr + HCR_STATUS_OFFSET)) >> 24); 237 238out: 239 up(&priv->cmd.poll_sem); 240 return err; 241} On Sat, Mar 7, 2009 at 12:55 AM, Dotan Barak wrote: > Please make sure that you have the latest firmware for the HCA that has the > failure. > > Dotan > > > Phillip Wilson wrote: > >> This is related to the thread I stared on Feb 24 >> >> Re: [ofa-general] ***SPAM*** Mellanox ibv_reg_mr (memory region) function >> call fails under load when using the mlx4 driver >> So far I have modified the "num_mtt" to 1 << 21, as Dotan suggested, but >> I will try some more values this weekend. >> I think the code for this is in ../drivers/net/mlx4/cmd.c mlx4_cmd( ... >> ) >> -ETIMEOUT > System Information: >> > >> > >> > >> > The system has 4GB of memory. >> > >> > >> > >> > uname -a >> > >> > Linux (none) 2.6.24.02.02.08 #21 SMP Thu Feb 19 11:04:35 PST 2009 ia64 >> > unknown >> > >> > >> > >> > OFED 1.2.5 >> > >> > >> > >> > lspci -d 15b3: >> > >> > >> > >> > 0000:10:00.0 InfiniBand: Mellanox Technologies MT25208 InfiniHost III Ex >> > (Tavor compatibility mode) (rev 20) >> > >> > 0000:c3:00.0 InfiniBand: Mellanox Technologies: Unknown device 634a (rev >> a0) >> > >> > >> > >> > lspci -d 15b3: -n >> > >> > 0000:10:00.0 0c06: 15b3:6278 (rev 20) >> > >> > 0000:c3:00.0 0c06: 15b3:634a (rev a0) >> > >> > >> > >> > ibv_devinfo -v >> > >> > hca_id: mlx4_0 >> > >> > fw_ver: 2.5.000 >> > >> > >> > >> > hca_id: mthca0 >> > >> > fw_ver: 4.8.930 >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From devel at morey-chaisemartin.com Sun Mar 8 00:45:53 2009 From: devel at morey-chaisemartin.com (Nicolas Morey-Chaisemartin) Date: Sun, 08 Mar 2009 09:45:53 +0100 Subject: [ofa-general] ***SPAM*** Re: mlx4_core 0000:c3:00.0: SW2HW_MPT failed (-16) (dmesg) In-Reply-To: <6e4f44220903080014x49591fc9idc21000b107f8771@mail.gmail.com> References: <6e4f44220903062016q8ca6d44ha22b10231ead5b3d@mail.gmail.com> <49B23680.1030508@gmail.com> <6e4f44220903080014x49591fc9idc21000b107f8771@mail.gmail.com> Message-ID: <49B385C1.7090104@morey-chaisemartin.com> Phillip Wilson a écrit : > I updated the HCA "InfiniBand: Mellanox Technologies: Unknown device > 634a (rev a0)" to the latest firmware and issue remains. "fw_ver" is > now 2.6.000. > > Any ideas on why the time out is occuring in the function? > > > I've seen this problem couples of times. Something on your system (probably HCA) is in a crappy state and won't answer. I had the problem (with top spin HCA and old OFED stacks on IA64 system). It was due to the driver trying to use msi_x and it failed. Try in your modprobe.conf file to force msi=0 msi_x=0 for mlx4_core module, it may help :) Nicolas From tziporet at dev.mellanox.co.il Sun Mar 8 04:30:03 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Sun, 08 Mar 2009 13:30:03 +0200 Subject: ***SPAM*** Re: [ofa-general] OFED 1.4.1 is available In-Reply-To: <92daa7bf0903051857v5754f52fjc0b2b10324ddd4ac@mail.gmail.com> References: <5D49E7A8952DC44FB38C38FA0D758EAD01F50B50@mtlexch01.mtl.com> <92daa7bf0903051857v5754f52fjc0b2b10324ddd4ac@mail.gmail.com> Message-ID: <49B3AC3B.5030605@mellanox.co.il> PN wrote: > Dear all, > > Sorry, I have another problem on installing the OFED-1.4.1-rc1 on > CentOS 5.2. > I can build all the rpm, but there is an error when installing the > mpitests_mvapich2_intel-3.1-891.x86_64.rpm > > .... > Install mpitests_mvapich_gcc RPM: > Running rpm -iv > /opt/software/packages/ofed/OFED-1.4.1-rc1/OFED-1.4.1-rc1/RPMS/centos-release-5-2.el5.centos/x86_64/mpitests_mvapich_gcc-3.1-891.x86_64.rpm > Install mpitests_mvapich_intel RPM: > Running rpm -iv > /opt/software/packages/ofed/OFED-1.4.1-rc1/OFED-1.4.1-rc1/RPMS/centos-release-5-2.el5.centos/x86_64/mpitests_mvapich_intel-3.1-891.x86_64.rpm > Install mpitests_mvapich2_gcc RPM: > Running rpm -iv > /opt/software/packages/ofed/OFED-1.4.1-rc1/OFED-1.4.1-rc1/RPMS/centos-release-5-2.el5.centos/x86_64/mpitests_mvapich2_gcc-3.1-891.x86_64.rpm > Install mpitests_mvapich2_intel RPM: > Running rpm -iv > /opt/software/packages/ofed/OFED-1.4.1-rc1/OFED-1.4.1-rc1/RPMS/centos-release-5-2.el5.centos/x86_64/mpitests_mvapich2_intel-3.1-891.x86_64.rpm > Failed to install mpitests_mvapich2_intel RPM > See /tmp/OFED.29515.logs/mpitests_mvapich2_intel.rpminstall.log > > > # cat /tmp/OFED.29515.logs/mpitests_mvapich2_intel.rpminstall.log > error: Failed dependencies: > libimf.so()(64bit) is needed by > mpitests_mvapich2_intel-3.1-891.x86_64 > libintlc.so.5()(64bit) is needed by > mpitests_mvapich2_intel-3.1-891.x86_64 > libsvml.so()(64bit) is needed by > mpitests_mvapich2_intel-3.1-891.x86_64 > > > I've installed Intel Compiler 11.0.081 and I found that all the > required .so are located in /opt/intel/Compiler/11.0/081/lib/intel64 > and the location is in LD_LIBRARY_PATH. > > # echo $LD_LIBRARY_PATH > /opt/intel/Compiler/11.0/081/lib/intel64:/opt/intel/Compiler/11.0/081/ipp/em64t/sharedlib:/opt/intel/Compiler/11.0/081/mkl/lib/em64t:/opt/intel/Compiler/11.0/081/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib:/opt/intel/Compiler/11.0/081/lib/intel64:/opt/intel/Compiler/11.0/081/ipp/em64t/sharedlib:/opt/intel/Compiler/11.0/081/mkl/lib/em64t:/opt/intel/Compiler/11.0/081/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib > > Any idea on how to solve this? > Thanks a lot. > > Regards, > PN Pasha Can you look at this? Thanks Tziporet From yevgenyp at mellanox.co.il Sun Mar 8 08:09:10 2009 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Sun, 08 Mar 2009 17:09:10 +0200 Subject: [ofa-general][PATCH 0/2 V2] mlx4: Link sensing support Message-ID: <49B3DF96.4070804@mellanox.co.il> This series of patches suggest the following changes comparing to the previous version: 1. Changed the order of patches because patch 2/2 suggested a fix to problem that could be caused by patch 1/2. 2. Changed the order of files in Makefile to be in alphabetic order. 3. United the work queues of the catastrophic error poller and the link sensing task. Now mlx4_core has a single work queue called "mlx4" 4. Using "cancel_delayed_task_sync()" to stop the sensing task. 5. Using "round_jiffies_relative()" instead of "round_jiffies()" Regards, Yevgeny From yevgenyp at mellanox.co.il Sun Mar 8 08:09:16 2009 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Sun, 08 Mar 2009 17:09:16 +0200 Subject: [ofa-general][PATCH 1/2 V2] mlx4_core: Don't perform SET_PORT command for Eth port from core Message-ID: <49B3DF9C.4080803@mellanox.co.il> The same operation is performed when the Ethernet driver initializes the port. Signed-off-by: Yevgeny Petrilin --- drivers/net/mlx4/port.c | 13 +++++-------- 1 files changed, 5 insertions(+), 8 deletions(-) diff --git a/drivers/net/mlx4/port.c b/drivers/net/mlx4/port.c index 0a057e5..7cce334 100644 --- a/drivers/net/mlx4/port.c +++ b/drivers/net/mlx4/port.c @@ -298,20 +298,17 @@ int mlx4_SET_PORT(struct mlx4_dev *dev, u8 port) { struct mlx4_cmd_mailbox *mailbox; int err; - u8 is_eth = dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH; mailbox = mlx4_alloc_cmd_mailbox(dev); if (IS_ERR(mailbox)) return PTR_ERR(mailbox); memset(mailbox->buf, 0, 256); - if (is_eth) { - ((u8 *) mailbox->buf)[3] = 6; - ((__be16 *) mailbox->buf)[4] = cpu_to_be16(1 << 15); - ((__be16 *) mailbox->buf)[6] = cpu_to_be16(1 << 15); - } else - ((__be32 *) mailbox->buf)[1] = dev->caps.ib_port_def_cap[port]; - err = mlx4_cmd(dev, mailbox->dma, port, is_eth, MLX4_CMD_SET_PORT, + if (dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH) + return 0; + + ((__be32 *) mailbox->buf)[1] = dev->caps.ib_port_def_cap[port]; + err = mlx4_cmd(dev, mailbox->dma, port, 0, MLX4_CMD_SET_PORT, MLX4_CMD_TIME_CLASS_B); mlx4_free_cmd_mailbox(dev, mailbox); -- 1.6.1.3 From yevgenyp at mellanox.co.il Sun Mar 8 08:09:24 2009 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Sun, 08 Mar 2009 17:09:24 +0200 Subject: [ofa-general][PATCH 2/2 V2] mlx4_core: Link sensing support Message-ID: <49B3DFA4.4050004@mellanox.co.il> >From 1e50e289d580f86859ea987cbf70f3381ca6413c Mon Sep 17 00:00:00 2001 From: Yevgeny Petrilin Date: Tue, 3 Mar 2009 13:29:50 +0200 Subject: [PATCH 2/2] mlx4_core: Link sensing support At any time when port link is down (except to driver restart), and port is configured to auto sensing, we try to sense port link type in order to determine how to initialize the port. If port type need to be changed, all mlx4 interfaces are unregistered and then registered again with the new port types. Sensing is done with intervals of 3 seconds. Signed-off-by: Yevgeny Petrilin --- drivers/net/mlx4/Makefile | 2 +- drivers/net/mlx4/catas.c | 16 +---- drivers/net/mlx4/eq.c | 16 +++-- drivers/net/mlx4/main.c | 102 ++++++++++++++++++++------- drivers/net/mlx4/mlx4.h | 27 +++++++- drivers/net/mlx4/sense.c | 159 +++++++++++++++++++++++++++++++++++++++++++ include/linux/mlx4/cmd.h | 1 + include/linux/mlx4/device.h | 6 +- 8 files changed, 278 insertions(+), 51 deletions(-) create mode 100644 drivers/net/mlx4/sense.c diff --git a/drivers/net/mlx4/Makefile b/drivers/net/mlx4/Makefile index a7a97bf..83b6f20 100644 --- a/drivers/net/mlx4/Makefile +++ b/drivers/net/mlx4/Makefile @@ -1,7 +1,7 @@ obj-$(CONFIG_MLX4_CORE) += mlx4_core.o mlx4_core-y := alloc.o catas.o cmd.o cq.o eq.o fw.o icm.o intf.o main.o mcg.o \ - mr.o pd.o port.o profile.o qp.o reset.o srq.o + mr.o pd.o port.o profile.o qp.o reset.o sense.o srq.o obj-$(CONFIG_MLX4_EN) += mlx4_en.o diff --git a/drivers/net/mlx4/catas.c b/drivers/net/mlx4/catas.c index f094ee0..aa9674b 100644 --- a/drivers/net/mlx4/catas.c +++ b/drivers/net/mlx4/catas.c @@ -42,7 +42,6 @@ enum { static DEFINE_SPINLOCK(catas_lock); static LIST_HEAD(catas_list); -static struct workqueue_struct *catas_wq; static struct work_struct catas_work; static int internal_err_reset = 1; @@ -77,7 +76,7 @@ static void poll_catas(unsigned long dev_ptr) list_add(&priv->catas_err.list, &catas_list); spin_unlock(&catas_lock); - queue_work(catas_wq, &catas_work); + queue_work(mlx4_wq, &catas_work); } } else mod_timer(&priv->catas_err.timer, @@ -146,18 +145,7 @@ void mlx4_stop_catas_poll(struct mlx4_dev *dev) spin_unlock_irq(&catas_lock); } -int __init mlx4_catas_init(void) +void __init mlx4_catas_init(void) { INIT_WORK(&catas_work, catas_reset); - - catas_wq = create_singlethread_workqueue("mlx4_err"); - if (!catas_wq) - return -ENOMEM; - - return 0; -} - -void mlx4_catas_cleanup(void) -{ - destroy_workqueue(catas_wq); } diff --git a/drivers/net/mlx4/eq.c b/drivers/net/mlx4/eq.c index 2c19bff..8830dcb 100644 --- a/drivers/net/mlx4/eq.c +++ b/drivers/net/mlx4/eq.c @@ -163,6 +163,7 @@ static int mlx4_eq_int(struct mlx4_dev *dev, struct mlx4_eq *eq) int cqn; int eqes_found = 0; int set_ci = 0; + int port; while ((eqe = next_eqe_sw(eq))) { /* @@ -203,11 +204,16 @@ static int mlx4_eq_int(struct mlx4_dev *dev, struct mlx4_eq *eq) break; case MLX4_EVENT_TYPE_PORT_CHANGE: - mlx4_dispatch_event(dev, - eqe->subtype == MLX4_PORT_CHANGE_SUBTYPE_ACTIVE ? - MLX4_DEV_EVENT_PORT_UP : - MLX4_DEV_EVENT_PORT_DOWN, - be32_to_cpu(eqe->event.port_change.port) >> 28); + port = be32_to_cpu(eqe->event.port_change.port) >> 28; + if (eqe->subtype == MLX4_PORT_CHANGE_SUBTYPE_DOWN) { + mlx4_dispatch_event(dev, MLX4_DEV_EVENT_PORT_DOWN, + port); + mlx4_priv(dev)->sense.do_sense_port[port] = 1; + } else { + mlx4_dispatch_event(dev, MLX4_DEV_EVENT_PORT_UP, + port); + mlx4_priv(dev)->sense.do_sense_port[port] = 0; + } break; case MLX4_EVENT_TYPE_CQ_ERROR: diff --git a/drivers/net/mlx4/main.c b/drivers/net/mlx4/main.c index 8480f03..e1e3b6c 100644 --- a/drivers/net/mlx4/main.c +++ b/drivers/net/mlx4/main.c @@ -51,6 +51,8 @@ MODULE_DESCRIPTION("Mellanox ConnectX HCA low-level driver"); MODULE_LICENSE("Dual BSD/GPL"); MODULE_VERSION(DRV_VERSION); +struct workqueue_struct *mlx4_wq; + #ifdef CONFIG_MLX4_DEBUG int mlx4_debug_level = 0; @@ -98,24 +100,23 @@ module_param_named(use_prio, use_prio, bool, 0444); MODULE_PARM_DESC(use_prio, "Enable steering by VLAN priority on ETH ports " "(0/1, default 0)"); -static int mlx4_check_port_params(struct mlx4_dev *dev, +int mlx4_check_port_params(struct mlx4_dev *dev, enum mlx4_port_type *port_type) { int i; for (i = 0; i < dev->caps.num_ports - 1; i++) { - if (port_type[i] != port_type[i+1] && - !(dev->caps.flags & MLX4_DEV_CAP_FLAG_DPDP)) { - mlx4_err(dev, "Only same port types supported " - "on this HCA, aborting.\n"); - return -EINVAL; + if (port_type[i] != port_type[i+1]) { + if (!(dev->caps.flags & MLX4_DEV_CAP_FLAG_DPDP)) { + mlx4_err(dev, "Only same port types supported " + "on this HCA, aborting.\n"); + return -EINVAL; + } + if ((port_type[i] == MLX4_PORT_TYPE_ETH) || + (port_type[i+1] == MLX4_PORT_TYPE_IB)) + return -EINVAL; } } - if ((port_type[0] == MLX4_PORT_TYPE_ETH) && - (port_type[1] == MLX4_PORT_TYPE_IB)) { - mlx4_err(dev, "eth-ib configuration is not supported.\n"); - return -EINVAL; - } for (i = 0; i < dev->caps.num_ports; i++) { if (!(port_type[i] & dev->caps.supported_type[i+1])) { @@ -225,6 +226,9 @@ static int mlx4_dev_cap(struct mlx4_dev *dev, struct mlx4_dev_cap *dev_cap) dev->caps.port_type[i] = MLX4_PORT_TYPE_IB; else dev->caps.port_type[i] = MLX4_PORT_TYPE_ETH; + dev->caps.possible_type[i] = dev->caps.port_type[i]; + mlx4_priv(dev)->sense.sense_allowed[i] = + dev->caps.supported_type[i] == MLX4_PORT_TYPE_AUTO ? 1 : 0; if (dev->caps.log_num_macs > dev_cap->log_max_macs[i]) { dev->caps.log_num_macs = dev_cap->log_max_macs[i]; @@ -263,7 +267,7 @@ static int mlx4_dev_cap(struct mlx4_dev *dev, struct mlx4_dev_cap *dev_cap) * Change the port configuration of the device. * Every user of this function must hold the port mutex. */ -static int mlx4_change_port_types(struct mlx4_dev *dev, +int mlx4_change_port_types(struct mlx4_dev *dev, enum mlx4_port_type *port_types) { int err = 0; @@ -271,6 +275,8 @@ static int mlx4_change_port_types(struct mlx4_dev *dev, int port; for (port = 0; port < dev->caps.num_ports; port++) { + /* Change the port type only if the new type is different + * from the current, and not set to Auto */ if (port_types[port] != dev->caps.port_type[port + 1]) { change = 1; dev->caps.port_type[port + 1] = port_types[port]; @@ -302,10 +308,17 @@ static ssize_t show_port_type(struct device *dev, struct mlx4_port_info *info = container_of(attr, struct mlx4_port_info, port_attr); struct mlx4_dev *mdev = info->dev; + char type[8]; + + sprintf(type, "%s", + (mdev->caps.port_type[info->port] == MLX4_PORT_TYPE_IB) ? + "ib" : "eth"); + if (mdev->caps.possible_type[info->port] == MLX4_PORT_TYPE_AUTO) + sprintf(buf, "auto (%s)\n", type); + else + sprintf(buf, "%s\n", type); - return sprintf(buf, "%s\n", - mdev->caps.port_type[info->port] == MLX4_PORT_TYPE_IB ? - "ib" : "eth"); + return strlen(buf); } static ssize_t set_port_type(struct device *dev, @@ -317,6 +330,7 @@ static ssize_t set_port_type(struct device *dev, struct mlx4_dev *mdev = info->dev; struct mlx4_priv *priv = mlx4_priv(mdev); enum mlx4_port_type types[MLX4_MAX_PORTS]; + enum mlx4_port_type new_types[MLX4_MAX_PORTS]; int i; int err = 0; @@ -324,26 +338,56 @@ static ssize_t set_port_type(struct device *dev, info->tmp_type = MLX4_PORT_TYPE_IB; else if (!strcmp(buf, "eth\n")) info->tmp_type = MLX4_PORT_TYPE_ETH; + else if (!strcmp(buf, "auto\n")) + info->tmp_type = MLX4_PORT_TYPE_AUTO; else { mlx4_err(mdev, "%s is not supported port type\n", buf); return -EINVAL; } + mlx4_stop_sense(mdev); mutex_lock(&priv->port_mutex); - for (i = 0; i < mdev->caps.num_ports; i++) + /* Possible type is always the one that was delivered */ + mdev->caps.possible_type[info->port] = info->tmp_type; + + for (i = 0; i < mdev->caps.num_ports; i++) { types[i] = priv->port[i+1].tmp_type ? priv->port[i+1].tmp_type : - mdev->caps.port_type[i+1]; + mdev->caps.possible_type[i+1]; + if (types[i] == MLX4_PORT_TYPE_AUTO) + types[i] = mdev->caps.port_type[i+1]; + } - err = mlx4_check_port_params(mdev, types); + if (!(mdev->caps.flags & MLX4_DEV_CAP_FLAG_DPDP)) { + for (i = 1; i <= mdev->caps.num_ports; i++) { + if (mdev->caps.possible_type[i] == MLX4_PORT_TYPE_AUTO) { + mdev->caps.possible_type[i] = mdev->caps.port_type[i]; + err = -EINVAL; + } + } + } + if (err) { + mlx4_err(mdev, "Auto sensing is not supported on this HCA. " + "Set only 'eth' or 'ib' for both ports " + "(should be the same)\n"); + goto out; + } + + mlx4_do_sense_ports(mdev, new_types, types); + + err = mlx4_check_port_params(mdev, new_types); if (err) goto out; - for (i = 1; i <= mdev->caps.num_ports; i++) - priv->port[i].tmp_type = 0; + /* We are about to apply the changes after the configuration + * was verified, no need to remember the temporary types + * any more */ + for (i = 0; i < mdev->caps.num_ports; i++) + priv->port[i + 1].tmp_type = 0; - err = mlx4_change_port_types(mdev, types); + err = mlx4_change_port_types(mdev, new_types); out: + mlx4_start_sense(mdev); mutex_unlock(&priv->port_mutex); return err ? err : count; } @@ -1117,6 +1161,9 @@ static int __mlx4_init_one(struct pci_dev *pdev, const struct pci_device_id *id) if (err) goto err_port; + mlx4_sense_init(dev); + mlx4_start_sense(dev); + pci_set_drvdata(pdev, dev); return 0; @@ -1182,13 +1229,12 @@ static void mlx4_remove_one(struct pci_dev *pdev) int p; if (dev) { + mlx4_stop_sense(dev); mlx4_unregister_device(dev); - for (p = 1; p <= dev->caps.num_ports; p++) { mlx4_cleanup_port_info(&priv->port[p]); mlx4_CLOSE_PORT(dev, p); } - mlx4_cleanup_mcg_table(dev); mlx4_cleanup_qp_table(dev); mlx4_cleanup_srq_table(dev); @@ -1266,9 +1312,11 @@ static int __init mlx4_init(void) if (mlx4_verify_params()) return -EINVAL; - ret = mlx4_catas_init(); - if (ret) - return ret; + mlx4_catas_init(); + + mlx4_wq = create_singlethread_workqueue("mlx4"); + if (!mlx4_wq) + return -ENOMEM; ret = pci_register_driver(&mlx4_driver); return ret < 0 ? ret : 0; @@ -1277,7 +1325,7 @@ static int __init mlx4_init(void) static void __exit mlx4_cleanup(void) { pci_unregister_driver(&mlx4_driver); - mlx4_catas_cleanup(); + destroy_workqueue(mlx4_wq); } module_init(mlx4_init); diff --git a/drivers/net/mlx4/mlx4.h b/drivers/net/mlx4/mlx4.h index e0213ba..5bd79c2 100644 --- a/drivers/net/mlx4/mlx4.h +++ b/drivers/net/mlx4/mlx4.h @@ -40,6 +40,7 @@ #include #include #include +#include #include #include @@ -276,6 +277,13 @@ struct mlx4_port_info { struct mlx4_vlan_table vlan_table; }; +struct mlx4_sense { + struct mlx4_dev *dev; + u8 do_sense_port[MLX4_MAX_PORTS + 1]; + u8 sense_allowed[MLX4_MAX_PORTS + 1]; + struct delayed_work sense_poll; +}; + struct mlx4_priv { struct mlx4_dev dev; @@ -305,6 +313,7 @@ struct mlx4_priv { struct mlx4_uar driver_uar; void __iomem *kar; struct mlx4_port_info port[MLX4_MAX_PORTS + 1]; + struct mlx4_sense sense; struct mutex port_mutex; }; @@ -313,6 +322,10 @@ static inline struct mlx4_priv *mlx4_priv(struct mlx4_dev *dev) return container_of(dev, struct mlx4_priv, dev); } +#define MLX4_SENSE_RANGE (HZ * 3) + +extern struct workqueue_struct *mlx4_wq; + u32 mlx4_bitmap_alloc(struct mlx4_bitmap *bitmap); void mlx4_bitmap_free(struct mlx4_bitmap *bitmap, u32 obj); u32 mlx4_bitmap_alloc_range(struct mlx4_bitmap *bitmap, int cnt, int align); @@ -346,8 +359,7 @@ void mlx4_cleanup_mcg_table(struct mlx4_dev *dev); void mlx4_start_catas_poll(struct mlx4_dev *dev); void mlx4_stop_catas_poll(struct mlx4_dev *dev); -int mlx4_catas_init(void); -void mlx4_catas_cleanup(void); +void mlx4_catas_init(void); int mlx4_restart_one(struct pci_dev *pdev); int mlx4_register_device(struct mlx4_dev *dev); void mlx4_unregister_device(struct mlx4_dev *dev); @@ -379,6 +391,17 @@ void mlx4_srq_event(struct mlx4_dev *dev, u32 srqn, int event_type); void mlx4_handle_catas_err(struct mlx4_dev *dev); +void mlx4_do_sense_ports(struct mlx4_dev *dev, + enum mlx4_port_type *stype, + enum mlx4_port_type *defaults); +void mlx4_start_sense(struct mlx4_dev *dev); +void mlx4_stop_sense(struct mlx4_dev *dev); +void mlx4_sense_init(struct mlx4_dev *dev); +int mlx4_check_port_params(struct mlx4_dev *dev, + enum mlx4_port_type *port_type); +int mlx4_change_port_types(struct mlx4_dev *dev, + enum mlx4_port_type *port_types); + void mlx4_init_mac_table(struct mlx4_dev *dev, struct mlx4_mac_table *table); void mlx4_init_vlan_table(struct mlx4_dev *dev, struct mlx4_vlan_table *table); diff --git a/drivers/net/mlx4/sense.c b/drivers/net/mlx4/sense.c new file mode 100644 index 0000000..341e656 --- /dev/null +++ b/drivers/net/mlx4/sense.c @@ -0,0 +1,159 @@ +/* + * Copyright (c) 2007 Mellanox Technologies. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ + +#include +#include + +#include + +#include "mlx4.h" + + +static int mlx4_SENSE_PORT(struct mlx4_dev *dev, int port, + enum mlx4_port_type *type) +{ + u64 out_param; + int err = 0; + + err = mlx4_cmd_imm(dev, 0, &out_param, port, 0, + MLX4_CMD_SENSE_PORT, MLX4_CMD_TIME_CLASS_B); + if (err) { + mlx4_err(dev, "Sense command failed for port: %d\n", port); + return err; + } + + if (out_param > 2) { + mlx4_err(dev, "Sense returned illegal value: 0x%llx\n", out_param); + return EINVAL; + } + + *type = out_param; + return 0; +} + +void mlx4_do_sense_ports(struct mlx4_dev *dev, + enum mlx4_port_type *stype, + enum mlx4_port_type *defaults) +{ + struct mlx4_sense *sense = &mlx4_priv(dev)->sense; + int err; + int i; + + for (i = 1; i <= dev->caps.num_ports; i++) { + stype[i-1] = 0; + if (sense->do_sense_port[i] && sense->sense_allowed[i] && + dev->caps.possible_type[i] == MLX4_PORT_TYPE_AUTO) { + err = mlx4_SENSE_PORT(dev, i, &stype[i-1]); + if (err) + stype[i-1] = defaults[i-1]; + } else + stype[i-1] = defaults[i-1]; + } + + /* + * Adjust port configuration: + * If port 1 sensed nothing and port 2 is IB, set both as IB + * If port 2 sensed nothing and port 1 is Eth, set both as Eth + */ + if (stype[0] == MLX4_PORT_TYPE_ETH) { + for (i = 1; i < dev->caps.num_ports; i++) + stype[i] = stype[i] ? stype[i] : MLX4_PORT_TYPE_ETH; + } + if (stype[dev->caps.num_ports - 1] == MLX4_PORT_TYPE_IB) { + for (i = 0; i < dev->caps.num_ports - 1; i++) + stype[i] = stype[i] ? stype[i] : MLX4_PORT_TYPE_IB; + } + + /* + * If sensed nothing, remain in current configuration. + */ + for (i = 0; i < dev->caps.num_ports; i++) + stype[i] = stype[i] ? stype[i] : defaults[i]; + +} + +static void mlx4_sense_port(struct work_struct *work) +{ + struct delayed_work *delay = container_of(work, struct delayed_work, work); + struct mlx4_sense *sense = container_of(delay, struct mlx4_sense, + sense_poll); + struct mlx4_dev *dev = sense->dev; + struct mlx4_priv *priv = mlx4_priv(dev); + enum mlx4_port_type stype[MLX4_MAX_PORTS]; + + mutex_lock(&priv->port_mutex); + mlx4_do_sense_ports(dev, stype, &dev->caps.port_type[1]); + + if (mlx4_check_port_params(dev, stype)) + goto sense_again; + + if (mlx4_change_port_types(dev, stype)) + mlx4_err(dev, "Failed to change port_types\n"); + +sense_again: + mutex_unlock(&priv->port_mutex); + queue_delayed_work(mlx4_wq , &sense->sense_poll, + round_jiffies_relative(MLX4_SENSE_RANGE)); +} + +void mlx4_start_sense(struct mlx4_dev *dev) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + struct mlx4_sense *sense = &priv->sense; + + if (!(dev->caps.flags & MLX4_DEV_CAP_FLAG_DPDP)) + return; + + queue_delayed_work(mlx4_wq , &sense->sense_poll, + round_jiffies_relative(MLX4_SENSE_RANGE)); +} + + +void mlx4_stop_sense(struct mlx4_dev *dev) +{ + cancel_delayed_work_sync(&mlx4_priv(dev)->sense.sense_poll); +} + +void mlx4_sense_init(struct mlx4_dev *dev) +{ + struct mlx4_priv *priv = mlx4_priv(dev); + struct mlx4_sense *sense = &priv->sense; + int port; + + sense->dev = dev; + for (port = 1; port <= dev->caps.num_ports; port++) + sense->do_sense_port[port] = 1; + + INIT_DELAYED_WORK_DEFERRABLE(&sense->sense_poll, mlx4_sense_port); +} + diff --git a/include/linux/mlx4/cmd.h b/include/linux/mlx4/cmd.h index cf9c679..0f82293 100644 --- a/include/linux/mlx4/cmd.h +++ b/include/linux/mlx4/cmd.h @@ -55,6 +55,7 @@ enum { MLX4_CMD_CLOSE_PORT = 0xa, MLX4_CMD_QUERY_HCA = 0xb, MLX4_CMD_QUERY_PORT = 0x43, + MLX4_CMD_SENSE_PORT = 0x4d, MLX4_CMD_SET_PORT = 0xc, MLX4_CMD_ACCESS_DDR = 0x2e, MLX4_CMD_MAP_ICM = 0xffa, diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h index 8f659cc..3aff8a6 100644 --- a/include/linux/mlx4/device.h +++ b/include/linux/mlx4/device.h @@ -155,8 +155,9 @@ enum mlx4_qp_region { }; enum mlx4_port_type { - MLX4_PORT_TYPE_IB = 1 << 0, - MLX4_PORT_TYPE_ETH = 1 << 1, + MLX4_PORT_TYPE_IB = 1, + MLX4_PORT_TYPE_ETH = 2, + MLX4_PORT_TYPE_AUTO = 3 }; enum mlx4_special_vlan_idx { @@ -237,6 +238,7 @@ struct mlx4_caps { enum mlx4_port_type port_type[MLX4_MAX_PORTS + 1]; u8 supported_type[MLX4_MAX_PORTS + 1]; u32 port_mask; + enum mlx4_port_type possible_type[MLX4_MAX_PORTS + 1]; }; struct mlx4_buf_list { -- 1.6.1.3 From dorfman.eli at gmail.com Mon Mar 9 01:13:58 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Mon, 09 Mar 2009 10:13:58 +0200 Subject: [ofa-general] ***SPAM*** Question: remove c99 definitions within the ib_mad_f structure Message-ID: <49B4CFC6.7020800@gmail.com> Hi, Can we revert commit 2dbb8b95d9dc27423a6fdb85d88ef385ecee0005 ? "remove c99 definitions within the ib_mad_f structure to enable better portability with WinOF." Using the definitions makes the code clearer and safer to maintain. Aren't there better alternative, such as using gcc for example? Thanks, Eli From dorfman.eli at gmail.com Mon Mar 9 02:38:32 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Mon, 09 Mar 2009 11:38:32 +0200 Subject: [ofa-general] ***SPAM*** opensm compilation fails Message-ID: <49B4E398.6050701@gmail.com> Hi Sasha, Commit 9b8561ac02e5e90c616b58bbbcfe2eb1a25d42fd breaks opensm: osm_console.o: In function `osm_console': /tmp/management/opensm/opensm/osm_console.c:1471: undefined reference to `cio_close' collect2: ld returned 1 exit status make[2]: *** [opensm] Error 1 make[2]: Leaving directory `/tmp/buildufm/management/opensm/opensm' make[1]: *** [all] Error 2 make[1]: Leaving directory `/tmp/buildufm/management/opensm/opensm' make: *** [all-recursive] Error 1 Eli From nicolas.morey-chaisemartin at ext.bull.net Mon Mar 9 02:45:23 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Mon, 09 Mar 2009 10:45:23 +0100 Subject: [ofa-general] Re: opensm compilation fails In-Reply-To: <49B4E398.6050701@gmail.com> References: <49B4E398.6050701@gmail.com> Message-ID: <49B4E533.4030208@ext.bull.net> Hi, I just checked again and opensm compiles without any problems. Try to run a make clean to be sure everything is up to date (some headers changed). If it still is a problem please send me some more logs, I'll look into it. Regards Nicolas Eli Dorfman (Voltaire) wrote: > Hi Sasha, > > Commit 9b8561ac02e5e90c616b58bbbcfe2eb1a25d42fd breaks opensm: > > osm_console.o: In function `osm_console': > /tmp/management/opensm/opensm/osm_console.c:1471: undefined reference to `cio_close' > collect2: ld returned 1 exit status > make[2]: *** [opensm] Error 1 > make[2]: Leaving directory `/tmp/buildufm/management/opensm/opensm' > make[1]: *** [all] Error 2 > make[1]: Leaving directory `/tmp/buildufm/management/opensm/opensm' > make: *** [all-recursive] Error 1 > > Eli > > From hnrose at comcast.net Mon Mar 9 04:01:17 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Mon, 9 Mar 2009 06:01:17 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] opensm/osm_inform.c: In __osm_send_report, make sure p_report_madw valid before using Message-ID: <20090309110117.GB18067@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_inform.c b/opensm/opensm/osm_inform.c index 6763a2a..f0dab30 100644 --- a/opensm/opensm/osm_inform.c +++ b/opensm/opensm/osm_inform.c @@ -326,8 +326,6 @@ static ib_api_status_t __osm_send_report(IN osm_infr_t * p_infr_rec, /* the info p_infr_rec->h_bind, MAD_BLOCK_SIZE, &(p_infr_rec->report_addr)); - p_report_madw->resp_expected = TRUE; - if (!p_report_madw) { OSM_LOG(p_log, OSM_LOG_ERROR, "ERR 0203" "osm_mad_pool_get failed\n"); @@ -335,6 +333,8 @@ static ib_api_status_t __osm_send_report(IN osm_infr_t * p_infr_rec, /* the info goto Exit; } + p_report_madw->resp_expected = TRUE; + /* advance trap trans id (cant simply ++ on some systems inside ntoh) */ p_mad = osm_madw_get_mad_ptr(p_report_madw); ib_mad_init_new(p_mad, IB_MCLASS_SUBN_ADM, 2, IB_MAD_METHOD_REPORT, From hnrose at comcast.net Mon Mar 9 04:00:20 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Mon, 9 Mar 2009 06:00:20 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] opensm/osm_req.c: Shouldn't reveal port's MKey on Trap method Message-ID: <20090309110020.GA18067@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c index 2e851b2..d5ce37a 100644 --- a/opensm/opensm/osm_req.c +++ b/opensm/opensm/osm_req.c @@ -257,7 +257,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) smp->method = IB_MAD_METHOD_TRAP; smp->trans_id = cl_hton64((uint64_t)cl_atomic_inc(&sm->sm_trans_id)); smp->attr_id = IB_MAD_ATTR_NOTICE; - smp->m_key = sm->p_subn->opt.m_key; + smp->m_key = 0; ntc = (ib_mad_notice_attr_t *)smp->data; From ogerlitz at voltaire.com Mon Mar 9 03:10:03 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Mon, 9 Mar 2009 12:10:03 +0200 (IST) Subject: [ofa-general] git head opensm build failure Message-ID: Hi Sasha, Attempting to build the head of the mng git tree, I come across the following error with opensm linking osm_console.o: In function `osm_console': opensm/opensm/osm_console.c:1471: undefined reference to `cio_close' from the code it seems like a call to cio_close which is not under ifdef ENABLE_OSM_CONSOLE_SOCKET Or. From vlad at lists.openfabrics.org Mon Mar 9 03:19:07 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Mon, 9 Mar 2009 03:19:07 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090309-0200 daily build status Message-ID: <20090309101908.167FDE61039@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From swise at opengridcomputing.com Mon Mar 9 03:20:14 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 09 Mar 2009 05:20:14 -0500 Subject: [ofa-general] [PATCH v3] RDMA/cxgb3: Handle EEH events for active connections. Message-ID: <20090309102014.5738.98445.stgit@build.ogc.int> - wrap calls into cxgb3 and fail them if we're in the middle of an eeh event. - correctly unwind and release endpoint and other resources when we are in an EEH event. - dispatch IB_EVENT_DEVICE_EVENT event when cxgb3 notifies iw_cxgb3 of a fatal error. Signed-off-by: Steve Wise --- drivers/infiniband/hw/cxgb3/cxio_hal.c | 10 ++-- drivers/infiniband/hw/cxgb3/cxio_hal.h | 6 ++ drivers/infiniband/hw/cxgb3/iwch.c | 11 +++- drivers/infiniband/hw/cxgb3/iwch.h | 5 ++ drivers/infiniband/hw/cxgb3/iwch_cm.c | 90 +++++++++++++++++++++++--------- drivers/infiniband/hw/cxgb3/iwch_qp.c | 4 + 6 files changed, 92 insertions(+), 34 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.c b/drivers/infiniband/hw/cxgb3/cxio_hal.c index 11efd35..630e3f4 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.c +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.c @@ -152,7 +152,7 @@ static int cxio_hal_clear_qp_ctx(struct cxio_rdev *rdev_p, u32 qpid) sge_cmd = qpid << 8 | 3; wqe->sge_cmd = cpu_to_be64(sge_cmd); skb->priority = CPL_PRIORITY_CONTROL; - return (cxgb3_ofld_send(rdev_p->t3cdev_p, skb)); + return iwch_cxgb3_ofld_send(rdev_p->t3cdev_p, skb); } int cxio_create_cq(struct cxio_rdev *rdev_p, struct t3_cq *cq) @@ -571,7 +571,7 @@ static int cxio_hal_init_ctrl_qp(struct cxio_rdev *rdev_p) (unsigned long long) rdev_p->ctrl_qp.dma_addr, rdev_p->ctrl_qp.workq, 1 << T3_CTRL_QP_SIZE_LOG2); skb->priority = CPL_PRIORITY_CONTROL; - return (cxgb3_ofld_send(rdev_p->t3cdev_p, skb)); + return iwch_cxgb3_ofld_send(rdev_p->t3cdev_p, skb); err: kfree_skb(skb); return err; @@ -701,7 +701,7 @@ static int __cxio_tpt_op(struct cxio_rdev *rdev_p, u32 reset_tpt_entry, u32 stag_idx; u32 wptr; - if (rdev_p->flags) + if (cxio_fatal_error(rdev_p)) return -EIO; stag_state = stag_state > 0; @@ -858,7 +858,7 @@ int cxio_rdma_init(struct cxio_rdev *rdev_p, struct t3_rdma_init_attr *attr) wqe->qp_dma_size = cpu_to_be32(attr->qp_dma_size); wqe->irs = cpu_to_be32(attr->irs); skb->priority = 0; /* 0=>ToeQ; 1=>CtrlQ */ - return (cxgb3_ofld_send(rdev_p->t3cdev_p, skb)); + return iwch_cxgb3_ofld_send(rdev_p->t3cdev_p, skb); } void cxio_register_ev_cb(cxio_hal_ev_callback_func_t ev_cb) @@ -1024,9 +1024,9 @@ void cxio_rdev_close(struct cxio_rdev *rdev_p) cxio_hal_pblpool_destroy(rdev_p); cxio_hal_rqtpool_destroy(rdev_p); list_del(&rdev_p->entry); - rdev_p->t3cdev_p->ulp = NULL; cxio_hal_destroy_ctrl_qp(rdev_p); cxio_hal_destroy_resource(rdev_p->rscp); + rdev_p->t3cdev_p->ulp = NULL; } } diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.h b/drivers/infiniband/hw/cxgb3/cxio_hal.h index 9ed65b0..2fd5d03 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.h +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.h @@ -112,6 +112,11 @@ struct cxio_rdev { #define CXIO_ERROR_FATAL 1 }; +static inline int cxio_fatal_error(struct cxio_rdev *rdev_p) +{ + return (rdev_p->flags & CXIO_ERROR_FATAL); +} + static inline int cxio_num_stags(struct cxio_rdev *rdev_p) { return min((int)T3_MAX_NUM_STAG, (int)((rdev_p->rnic_info.tpt_top - rdev_p->rnic_info.tpt_base) >> 5)); @@ -185,6 +190,7 @@ void cxio_count_scqes(struct t3_cq *cq, struct t3_wq *wq, int *count); void cxio_flush_hw_cq(struct t3_cq *cq); int cxio_poll_cq(struct t3_wq *wq, struct t3_cq *cq, struct t3_cqe *cqe, u8 *cqe_flushed, u64 *cookie, u32 *credit); +int iwch_cxgb3_ofld_send(struct t3cdev *tdev, struct sk_buff *skb); #define MOD "iw_cxgb3: " #define PDBG(fmt, args...) pr_debug(MOD fmt, ## args) diff --git a/drivers/infiniband/hw/cxgb3/iwch.c b/drivers/infiniband/hw/cxgb3/iwch.c index 37a4fc2..26fc0a4 100644 --- a/drivers/infiniband/hw/cxgb3/iwch.c +++ b/drivers/infiniband/hw/cxgb3/iwch.c @@ -165,12 +165,19 @@ static void close_rnic_dev(struct t3cdev *tdev) static void iwch_err_handler(struct t3cdev *tdev, u32 status, u32 error) { struct cxio_rdev *rdev = tdev->ulp; + struct iwch_dev *rnicp = rdev_to_iwch_dev(rdev); + struct ib_event event; - if (status == OFFLOAD_STATUS_DOWN) + if (status == OFFLOAD_STATUS_DOWN) { rdev->flags = CXIO_ERROR_FATAL; - return; + event.device = &rnicp->ibdev; + event.event = IB_EVENT_DEVICE_FATAL; + event.element.port_num = 0; + ib_dispatch_event(&event); + } + return; } static int __init iwch_init_module(void) diff --git a/drivers/infiniband/hw/cxgb3/iwch.h b/drivers/infiniband/hw/cxgb3/iwch.h index 3773453..8473550 100644 --- a/drivers/infiniband/hw/cxgb3/iwch.h +++ b/drivers/infiniband/hw/cxgb3/iwch.h @@ -117,6 +117,11 @@ static inline struct iwch_dev *to_iwch_dev(struct ib_device *ibdev) return container_of(ibdev, struct iwch_dev, ibdev); } +static inline struct iwch_dev *rdev_to_iwch_dev(struct cxio_rdev *rdev) +{ + return container_of(rdev, struct iwch_dev, rdev); +} + static inline int t3b_device(const struct iwch_dev *rhp) { return rhp->rdev.t3cdev_p->type == T3B; diff --git a/drivers/infiniband/hw/cxgb3/iwch_cm.c b/drivers/infiniband/hw/cxgb3/iwch_cm.c index 44e936e..7f57872 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_cm.c +++ b/drivers/infiniband/hw/cxgb3/iwch_cm.c @@ -139,6 +139,38 @@ static void stop_ep_timer(struct iwch_ep *ep) put_ep(&ep->com); } +int iwch_l2t_send(struct t3cdev *tdev, struct sk_buff *skb, struct l2t_entry *l2e) +{ + int error=0; + struct cxio_rdev *rdev; + + rdev = (struct cxio_rdev *)tdev->ulp; + if (cxio_fatal_error(rdev)) { + kfree_skb(skb); + return -EIO; + } + error = l2t_send(tdev, skb, l2e); + if (error) + kfree_skb(skb); + return error; +} + +int iwch_cxgb3_ofld_send(struct t3cdev *tdev, struct sk_buff *skb) +{ + int error=0; + struct cxio_rdev *rdev; + + rdev = (struct cxio_rdev *)tdev->ulp; + if (cxio_fatal_error(rdev)) { + kfree_skb(skb); + return -EIO; + } + error = cxgb3_ofld_send(tdev, skb); + if (error) + kfree_skb(skb); + return error; +} + static void release_tid(struct t3cdev *tdev, u32 hwtid, struct sk_buff *skb) { struct cpl_tid_release *req; @@ -150,7 +182,7 @@ static void release_tid(struct t3cdev *tdev, u32 hwtid, struct sk_buff *skb) req->wr.wr_hi = htonl(V_WR_OP(FW_WROPCODE_FORWARD)); OPCODE_TID(req) = htonl(MK_OPCODE_TID(CPL_TID_RELEASE, hwtid)); skb->priority = CPL_PRIORITY_SETUP; - cxgb3_ofld_send(tdev, skb); + iwch_cxgb3_ofld_send(tdev, skb); return; } @@ -172,8 +204,7 @@ int iwch_quiesce_tid(struct iwch_ep *ep) req->val = cpu_to_be64(1 << S_TCB_RX_QUIESCE); skb->priority = CPL_PRIORITY_DATA; - cxgb3_ofld_send(ep->com.tdev, skb); - return 0; + return iwch_cxgb3_ofld_send(ep->com.tdev, skb); } int iwch_resume_tid(struct iwch_ep *ep) @@ -194,8 +225,7 @@ int iwch_resume_tid(struct iwch_ep *ep) req->val = 0; skb->priority = CPL_PRIORITY_DATA; - cxgb3_ofld_send(ep->com.tdev, skb); - return 0; + return iwch_cxgb3_ofld_send(ep->com.tdev, skb); } static void set_emss(struct iwch_ep *ep, u16 opt) @@ -382,7 +412,7 @@ static void abort_arp_failure(struct t3cdev *dev, struct sk_buff *skb) PDBG("%s t3cdev %p\n", __func__, dev); req->cmd = CPL_ABORT_NO_RST; - cxgb3_ofld_send(dev, skb); + iwch_cxgb3_ofld_send(dev, skb); } static int send_halfclose(struct iwch_ep *ep, gfp_t gfp) @@ -402,8 +432,7 @@ static int send_halfclose(struct iwch_ep *ep, gfp_t gfp) req->wr.wr_hi = htonl(V_WR_OP(FW_WROPCODE_OFLD_CLOSE_CON)); req->wr.wr_lo = htonl(V_WR_TID(ep->hwtid)); OPCODE_TID(req) = htonl(MK_OPCODE_TID(CPL_CLOSE_CON_REQ, ep->hwtid)); - l2t_send(ep->com.tdev, skb, ep->l2t); - return 0; + return iwch_l2t_send(ep->com.tdev, skb, ep->l2t); } static int send_abort(struct iwch_ep *ep, struct sk_buff *skb, gfp_t gfp) @@ -424,8 +453,7 @@ static int send_abort(struct iwch_ep *ep, struct sk_buff *skb, gfp_t gfp) req->wr.wr_lo = htonl(V_WR_TID(ep->hwtid)); OPCODE_TID(req) = htonl(MK_OPCODE_TID(CPL_ABORT_REQ, ep->hwtid)); req->cmd = CPL_ABORT_SEND_RST; - l2t_send(ep->com.tdev, skb, ep->l2t); - return 0; + return iwch_l2t_send(ep->com.tdev, skb, ep->l2t); } static int send_connect(struct iwch_ep *ep) @@ -469,8 +497,7 @@ static int send_connect(struct iwch_ep *ep) req->opt0l = htonl(opt0l); req->params = 0; req->opt2 = htonl(opt2); - l2t_send(ep->com.tdev, skb, ep->l2t); - return 0; + return iwch_l2t_send(ep->com.tdev, skb, ep->l2t); } static void send_mpa_req(struct iwch_ep *ep, struct sk_buff *skb) @@ -527,7 +554,7 @@ static void send_mpa_req(struct iwch_ep *ep, struct sk_buff *skb) req->sndseq = htonl(ep->snd_seq); BUG_ON(ep->mpa_skb); ep->mpa_skb = skb; - l2t_send(ep->com.tdev, skb, ep->l2t); + iwch_l2t_send(ep->com.tdev, skb, ep->l2t); start_ep_timer(ep); state_set(&ep->com, MPA_REQ_SENT); return; @@ -578,8 +605,7 @@ static int send_mpa_reject(struct iwch_ep *ep, const void *pdata, u8 plen) req->sndseq = htonl(ep->snd_seq); BUG_ON(ep->mpa_skb); ep->mpa_skb = skb; - l2t_send(ep->com.tdev, skb, ep->l2t); - return 0; + return iwch_l2t_send(ep->com.tdev, skb, ep->l2t); } static int send_mpa_reply(struct iwch_ep *ep, const void *pdata, u8 plen) @@ -630,8 +656,7 @@ static int send_mpa_reply(struct iwch_ep *ep, const void *pdata, u8 plen) req->sndseq = htonl(ep->snd_seq); ep->mpa_skb = skb; state_set(&ep->com, MPA_REP_SENT); - l2t_send(ep->com.tdev, skb, ep->l2t); - return 0; + return iwch_l2t_send(ep->com.tdev, skb, ep->l2t); } static int act_establish(struct t3cdev *tdev, struct sk_buff *skb, void *ctx) @@ -795,7 +820,7 @@ static int update_rx_credits(struct iwch_ep *ep, u32 credits) OPCODE_TID(req) = htonl(MK_OPCODE_TID(CPL_RX_DATA_ACK, ep->hwtid)); req->credit_dack = htonl(V_RX_CREDITS(credits) | V_RX_FORCE_ACK(1)); skb->priority = CPL_PRIORITY_ACK; - cxgb3_ofld_send(ep->com.tdev, skb); + iwch_cxgb3_ofld_send(ep->com.tdev, skb); return credits; } @@ -1203,8 +1228,7 @@ static int listen_start(struct iwch_listen_ep *ep) req->opt1 = htonl(V_CONN_POLICY(CPL_CONN_POLICY_ASK)); skb->priority = 1; - cxgb3_ofld_send(ep->com.tdev, skb); - return 0; + return iwch_cxgb3_ofld_send(ep->com.tdev, skb); } static int pass_open_rpl(struct t3cdev *tdev, struct sk_buff *skb, void *ctx) @@ -1237,8 +1261,7 @@ static int listen_stop(struct iwch_listen_ep *ep) req->cpu_idx = 0; OPCODE_TID(req) = htonl(MK_OPCODE_TID(CPL_CLOSE_LISTSRV_REQ, ep->stid)); skb->priority = 1; - cxgb3_ofld_send(ep->com.tdev, skb); - return 0; + return iwch_cxgb3_ofld_send(ep->com.tdev, skb); } static int close_listsrv_rpl(struct t3cdev *tdev, struct sk_buff *skb, @@ -1286,7 +1309,7 @@ static void accept_cr(struct iwch_ep *ep, __be32 peer_ip, struct sk_buff *skb) rpl->opt2 = htonl(opt2); rpl->rsvd = rpl->opt2; /* workaround for HW bug */ skb->priority = CPL_PRIORITY_SETUP; - l2t_send(ep->com.tdev, skb, ep->l2t); + iwch_l2t_send(ep->com.tdev, skb, ep->l2t); return; } @@ -1315,7 +1338,7 @@ static void reject_cr(struct t3cdev *tdev, u32 hwtid, __be32 peer_ip, rpl->opt0l_status = htonl(CPL_PASS_OPEN_REJECT); rpl->opt2 = 0; rpl->rsvd = rpl->opt2; - cxgb3_ofld_send(tdev, skb); + iwch_cxgb3_ofld_send(tdev, skb); } } @@ -1613,7 +1636,7 @@ static int peer_abort(struct t3cdev *tdev, struct sk_buff *skb, void *ctx) rpl->wr.wr_lo = htonl(V_WR_TID(ep->hwtid)); OPCODE_TID(rpl) = htonl(MK_OPCODE_TID(CPL_ABORT_RPL, ep->hwtid)); rpl->cmd = CPL_ABORT_NO_RST; - cxgb3_ofld_send(ep->com.tdev, rpl_skb); + iwch_cxgb3_ofld_send(ep->com.tdev, rpl_skb); out: if (release) release_ep_resources(ep); @@ -2014,8 +2037,11 @@ int iwch_destroy_listen(struct iw_cm_id *cm_id) ep->com.rpl_done = 0; ep->com.rpl_err = 0; err = listen_stop(ep); + if (err) + goto done; wait_event(ep->com.waitq, ep->com.rpl_done); cxgb3_free_stid(ep->com.tdev, ep->stid); +done: err = ep->com.rpl_err; cm_id->rem_ref(cm_id); put_ep(&ep->com); @@ -2027,12 +2053,22 @@ int iwch_ep_disconnect(struct iwch_ep *ep, int abrupt, gfp_t gfp) int ret=0; unsigned long flags; int close = 0; + int fatal = 0; + struct t3cdev *tdev; + struct cxio_rdev *rdev; spin_lock_irqsave(&ep->com.lock, flags); PDBG("%s ep %p state %s, abrupt %d\n", __func__, ep, states[ep->com.state], abrupt); + tdev = (struct t3cdev *)ep->com.tdev; + rdev = (struct cxio_rdev *)tdev->ulp; + if (cxio_fatal_error(rdev)) { + fatal = 1; + close_complete_upcall(ep); + ep->com.state = DEAD; + } switch (ep->com.state) { case MPA_REQ_WAIT: case MPA_REQ_SENT: @@ -2072,7 +2108,11 @@ int iwch_ep_disconnect(struct iwch_ep *ep, int abrupt, gfp_t gfp) ret = send_abort(ep, NULL, gfp); else ret = send_halfclose(ep, gfp); + if (ret) + fatal = 1; } + if (fatal) + release_ep_resources(ep); return ret; } diff --git a/drivers/infiniband/hw/cxgb3/iwch_qp.c b/drivers/infiniband/hw/cxgb3/iwch_qp.c index 19661b2..47957e1 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_qp.c +++ b/drivers/infiniband/hw/cxgb3/iwch_qp.c @@ -754,7 +754,7 @@ int iwch_post_zb_read(struct iwch_qp *qhp) wqe->send.wrh.gen_tid_len = cpu_to_be32(V_FW_RIWR_TID(qhp->ep->hwtid)| V_FW_RIWR_LEN(flit_cnt)); skb->priority = CPL_PRIORITY_DATA; - return cxgb3_ofld_send(qhp->rhp->rdev.t3cdev_p, skb); + return iwch_cxgb3_ofld_send(qhp->rhp->rdev.t3cdev_p, skb); } /* @@ -786,7 +786,7 @@ int iwch_post_terminate(struct iwch_qp *qhp, struct respQ_msg_t *rsp_msg) V_FW_RIWR_FLAGS(T3_COMPLETION_FLAG | T3_NOTIFY_FLAG)); wqe->send.wrh.gen_tid_len = cpu_to_be32(V_FW_RIWR_TID(qhp->ep->hwtid)); skb->priority = CPL_PRIORITY_DATA; - return cxgb3_ofld_send(qhp->rhp->rdev.t3cdev_p, skb); + return iwch_cxgb3_ofld_send(qhp->rhp->rdev.t3cdev_p, skb); } /* From swise at opengridcomputing.com Mon Mar 9 03:21:15 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 09 Mar 2009 05:21:15 -0500 Subject: [ofa-general] [PATCH 2.6.30] RDMA/cxgb3: Enforce required firmware during device discovery. Message-ID: <20090309102114.5779.32122.stgit@build.ogc.int> The cxgb3 NIC driver no longer enforces this, so iw_cxgb3 must. The FW major number indicates a specific interface between the FW and iw_cxgb3. Thus if the major number of the running firmware does not match the required version compiled into iw_cxgb3, then iw_cxgb3 must not register that device. Signed-off-by: Steve Wise --- drivers/infiniband/hw/cxgb3/cxio_hal.c | 17 +++++++++++++++++ drivers/infiniband/hw/cxgb3/cxio_hal.h | 3 +++ 2 files changed, 20 insertions(+), 0 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.c b/drivers/infiniband/hw/cxgb3/cxio_hal.c index 630e3f4..7a355d6 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.c +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.c @@ -941,6 +941,23 @@ int cxio_rdev_open(struct cxio_rdev *rdev_p) if (!rdev_p->t3cdev_p) rdev_p->t3cdev_p = dev2t3cdev(netdev_p); rdev_p->t3cdev_p->ulp = (void *) rdev_p; + + err = rdev_p->t3cdev_p->ctl(rdev_p->t3cdev_p, GET_EMBEDDED_INFO, + &(rdev_p->fw_info)); + if (err) { + printk(KERN_ERR "%s t3cdev_p(%p)->ctl returned error %d.\n", + __func__, rdev_p->t3cdev_p, err); + goto err1; + } + if (G_FW_VERSION_MAJOR(rdev_p->fw_info.fw_vers) != CXIO_FW_MAJ) { + printk(KERN_ERR MOD "fatal firmware version mismatch: " + "need version %u but adapter has version %u\n", + CXIO_FW_MAJ, + G_FW_VERSION_MAJOR(rdev_p->fw_info.fw_vers)); + err = -EINVAL; + goto err1; + } + err = rdev_p->t3cdev_p->ctl(rdev_p->t3cdev_p, RDMA_GET_PARAMS, &(rdev_p->rnic_info)); if (err) { diff --git a/drivers/infiniband/hw/cxgb3/cxio_hal.h b/drivers/infiniband/hw/cxgb3/cxio_hal.h index 2fd5d03..4399b04 100644 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.h +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.h @@ -61,6 +61,8 @@ #define T3_MAX_DEV_NAME_LEN 32 +#define CXIO_FW_MAJ 7 + struct cxio_hal_ctrl_qp { u32 wptr; u32 rptr; @@ -110,6 +112,7 @@ struct cxio_rdev { struct list_head entry; u32 flags; #define CXIO_ERROR_FATAL 1 + struct ch_embedded_info fw_info; }; static inline int cxio_fatal_error(struct cxio_rdev *rdev_p) From nicolas.morey-chaisemartin at ext.bull.net Mon Mar 9 03:30:36 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Mon, 09 Mar 2009 11:30:36 +0100 Subject: [ofa-general] [PATCH] Fixed cio_close use when ENABLE_OSM_CONSOLE_SOCKET is not set Message-ID: <49B4EFCC.2050404@ext.bull.net> Fixes compilation problem introduced by 9b8561ac02e5e90c616b58bbbcfe2eb1a25d42fd opensm/console: Fixed osm_console poll to handle POLLHUP Signed-off-by: Nicolas Morey-Chaisemartin --- Sorry for that, I have build script which always compiles with --enable-socket so I missed this one. opensm/opensm/osm_console.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/opensm/opensm/osm_console.c b/opensm/opensm/osm_console.c index 04c820c..0986fcc 100644 --- a/opensm/opensm/osm_console.c +++ b/opensm/opensm/osm_console.c @@ -1466,11 +1466,13 @@ int osm_console(osm_opensm_t * p_osm) } /* input fd is closed (hanged up) */ if (pollfd[1].revents & POLLHUP) { +#ifdef ENABLE_OSM_CONSOLE_SOCKET /* If we are using a sowket, we close the current connection */ if (p_oct->socket >= 0) { cio_close(p_oct); return 0; } +#endif /* If we use a local console, stdin is closed (most probable is pipe ended) * so we close the local console */ return -1; -- 1.6.2-rc2.GIT From acceptany at gmail.com Mon Mar 9 04:41:05 2009 From: acceptany at gmail.com (Jordan) Date: Mon, 9 Mar 2009 19:41:05 +0800 Subject: [ofa-general] ***SPAM*** problem about the OpenSM Message-ID: <91fe68d50903090441t114d70abv2f5091d00d3d185b@mail.gmail.com> I want to do some research about the routing algorithm in the OpenSM, for example, adding a new routing algorithm (e.g for torus or mesh)or improving some of them. But, after re-configuring and rebuilding the OpenSM, can the OpenSM still be installed on Mellanox switch? Can the new routing algorithm take effect in the network? What steps can I take to achieve this purpose? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hnrose at comcast.net Mon Mar 9 06:54:01 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Mon, 9 Mar 2009 08:54:01 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] libibmad/rpc.c: Handle redirection status Message-ID: <20090309135401.GA21411@comcast.net> Also, in mad_rpc, status should be based on management class Signed-off-by: Hal Rosenstock --- diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c index 210f0c2..c18b086 100644 --- a/libibmad/src/rpc.c +++ b/libibmad/src/rpc.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2006 Voltaire Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -168,7 +169,7 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport void *payload, void *rcvdata) { int status, len; - uint8_t sndbuf[1024], rcvbuf[1024], *mad; + uint8_t sndbuf[1024], rcvbuf[1024], *mad, mgmtclass; len = 0; memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); @@ -185,10 +186,28 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport mad = umad_get_mad(rcvbuf); - if ((status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F)) != 0) { - ERRS("MAD completed with error status 0x%x; dport (%s)", - status, portid2str(dport)); - return 0; + mgmtclass = mad_get_field(mad, 0, IB_MAD_MGMTCLASS_F); + if (mgmtclass == 1 || mgmtclass == 0x81) { + if (mgmtclass == 1) + status = mad_get_field(mad, 0, IB_MAD_STATUS_F); + else + status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F); + if (status != 0) { + ERRS("MAD completed with error status 0x%x; dport (%s)", + status, portid2str(dport)); + return 0; + } + } else { + if ((status = mad_get_field(mad, 0, IB_MAD_STATUS_F)) != 0) { + if (status & 2) { /* redirection */ + ERRS("MAD redirection not supported; dport (%s)", + portid2str(dport)); + } else { + ERRS("MAD completed with error status 0x%x; dport (%s)", + status, portid2str(dport)); + } + return 0; + } } if (ibdebug) { @@ -225,8 +244,13 @@ void *mad_rpc_rmpp(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * mad = umad_get_mad(rcvbuf); if ((status = mad_get_field(mad, 0, IB_MAD_STATUS_F)) != 0) { - ERRS("MAD completed with error status 0x%x; dport (%s)", - status, portid2str(dport)); + if (status & 2) { /* redirection */ + ERRS("MAD redirection not supported; dport (%s)", + portid2str(dport)); + } else { + ERRS("MAD completed with error status 0x%x; dport (%s)", + status, portid2str(dport)); + } return 0; } From sashak at voltaire.com Mon Mar 9 06:58:26 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Mar 2009 15:58:26 +0200 Subject: [ofa-general] Re: opensm compilation fails In-Reply-To: <49B4E533.4030208@ext.bull.net> References: <49B4E398.6050701@gmail.com> <49B4E533.4030208@ext.bull.net> Message-ID: <20090309135826.GF15398@sashak.voltaire.com> On 10:45 Mon 09 Mar , Nicolas Morey Chaisemartin wrote: > I just checked again and opensm compiles without any problems. I think Eli is using './configure --disable-console-socket' (which is default btw). And yes, there is an obvious error. I'm fixing... Sasha From kliteyn at dev.mellanox.co.il Mon Mar 9 06:56:59 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 09 Mar 2009 15:56:59 +0200 Subject: [ofa-general] Re: opensm compilation fails In-Reply-To: <20090309135826.GF15398@sashak.voltaire.com> References: <49B4E398.6050701@gmail.com> <49B4E533.4030208@ext.bull.net> <20090309135826.GF15398@sashak.voltaire.com> Message-ID: <49B5202B.5040706@dev.mellanox.co.il> Hi Sasha, Sasha Khapyorsky wrote: > On 10:45 Mon 09 Mar , Nicolas Morey Chaisemartin wrote: >> I just checked again and opensm compiles without any problems. > > I think Eli is using './configure --disable-console-socket' (which is > default btw). And yes, there is an obvious error. I'm fixing... Nicolas has already posted a fix -- Yevgeny > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From sashak at voltaire.com Mon Mar 9 07:12:03 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Mar 2009 16:12:03 +0200 Subject: [ofa-general] [PATCH] opensm: fix build failure with --disable-console-socket In-Reply-To: <20090309135826.GF15398@sashak.voltaire.com> References: <49B4E398.6050701@gmail.com> <49B4E533.4030208@ext.bull.net> <20090309135826.GF15398@sashak.voltaire.com> Message-ID: <20090309141203.GG15398@sashak.voltaire.com> Fix OpenSM build failure - when configured with '--disable-console-socket' cio_close() becomes undefined and console_close() unused. Signed-off-by: Sasha Khapyorsky --- opensm/include/opensm/osm_console_io.h | 2 ++ opensm/opensm/osm_console_io.c | 5 ----- 2 files changed, 2 insertions(+), 5 deletions(-) diff --git a/opensm/include/opensm/osm_console_io.h b/opensm/include/opensm/osm_console_io.h index f31d811..91f341c 100644 --- a/opensm/include/opensm/osm_console_io.h +++ b/opensm/include/opensm/osm_console_io.h @@ -85,6 +85,8 @@ int is_console_enabled(osm_subn_opt_t *p_opt); int cio_open(osm_console_t * p_oct, int new_fd, osm_log_t * p_log); int cio_close(osm_console_t * p_oct); int is_authorized(osm_console_t * p_oct); +#else +#define cio_close(oct) (0) #endif END_C_DECLS diff --git a/opensm/opensm/osm_console_io.c b/opensm/opensm/osm_console_io.c index b0a8356..606b6f1 100644 --- a/opensm/opensm/osm_console_io.c +++ b/opensm/opensm/osm_console_io.c @@ -106,12 +106,10 @@ int cio_close(osm_console_t * p_oct) } return rtnval; } -#endif /* close the connection */ static void console_close(osm_console_t * p_oct, osm_log_t * p_log) { -#ifdef ENABLE_OSM_CONSOLE_SOCKET if ((p_oct->socket > 0) && (p_oct->in_fd != -1)) { OSM_LOG(p_log, OSM_LOG_INFO, "Console connection closed: %s (%s)\n", @@ -122,14 +120,11 @@ static void console_close(osm_console_t * p_oct, osm_log_t * p_log) close(p_oct->socket); p_oct->socket = -1; } -#endif } - /********************************************************************** * Do authentication & authorization check **********************************************************************/ -#ifdef ENABLE_OSM_CONSOLE_SOCKET int is_authorized(osm_console_t * p_oct) { /* allowed to use the console? */ -- 1.6.1.2.319.gbd9e From sashak at voltaire.com Mon Mar 9 07:23:10 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Mar 2009 16:23:10 +0200 Subject: [ofa-general] Re: opensm compilation fails In-Reply-To: <49B5202B.5040706@dev.mellanox.co.il> References: <49B4E398.6050701@gmail.com> <49B4E533.4030208@ext.bull.net> <20090309135826.GF15398@sashak.voltaire.com> <49B5202B.5040706@dev.mellanox.co.il> Message-ID: <20090309142303.GH15398@sashak.voltaire.com> On 15:56 Mon 09 Mar , Yevgeny Kliteynik wrote: > > Nicolas has already posted a fix I see now. My patch also fixes warning (about unused console_close()). Will apply both :). Sasha From sashak at voltaire.com Mon Mar 9 07:24:53 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Mar 2009 16:24:53 +0200 Subject: [ofa-general] [PATCH v2] opensm: fix build warning with --disable-console-socket In-Reply-To: <20090309142303.GH15398@sashak.voltaire.com> References: <49B4E398.6050701@gmail.com> <49B4E533.4030208@ext.bull.net> <20090309135826.GF15398@sashak.voltaire.com> <49B5202B.5040706@dev.mellanox.co.il> <20090309142303.GH15398@sashak.voltaire.com> Message-ID: <20090309142453.GI15398@sashak.voltaire.com> Fix OpenSM build warning - when configured with '--disable-console-socket' console_close() ibecomes unused. Signed-off-by: Sasha Khapyorsky --- opensm/opensm/osm_console_io.c | 5 ----- 1 files changed, 0 insertions(+), 5 deletions(-) diff --git a/opensm/opensm/osm_console_io.c b/opensm/opensm/osm_console_io.c index b0a8356..606b6f1 100644 --- a/opensm/opensm/osm_console_io.c +++ b/opensm/opensm/osm_console_io.c @@ -106,12 +106,10 @@ int cio_close(osm_console_t * p_oct) } return rtnval; } -#endif /* close the connection */ static void console_close(osm_console_t * p_oct, osm_log_t * p_log) { -#ifdef ENABLE_OSM_CONSOLE_SOCKET if ((p_oct->socket > 0) && (p_oct->in_fd != -1)) { OSM_LOG(p_log, OSM_LOG_INFO, "Console connection closed: %s (%s)\n", @@ -122,14 +120,11 @@ static void console_close(osm_console_t * p_oct, osm_log_t * p_log) close(p_oct->socket); p_oct->socket = -1; } -#endif } - /********************************************************************** * Do authentication & authorization check **********************************************************************/ -#ifdef ENABLE_OSM_CONSOLE_SOCKET int is_authorized(osm_console_t * p_oct) { /* allowed to use the console? */ -- 1.6.1.2.319.gbd9e From sashak at voltaire.com Mon Mar 9 07:29:22 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Mar 2009 16:29:22 +0200 Subject: [ofa-general] Re: opensm compilation fails In-Reply-To: <49B5202B.5040706@dev.mellanox.co.il> References: <49B4E398.6050701@gmail.com> <49B4E533.4030208@ext.bull.net> <20090309135826.GF15398@sashak.voltaire.com> <49B5202B.5040706@dev.mellanox.co.il> Message-ID: <20090309142914.GJ15398@sashak.voltaire.com> Yevgeny, BTW, could we do something with this warning (gcc-4.3.3)? osm_qos_parser_l.c: In function 'yylex': osm_qos_parser_l.l:358: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result Sasha From sashak at voltaire.com Mon Mar 9 07:47:06 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Mar 2009 16:47:06 +0200 Subject: [ofa-general] Re: [PATCH] Fixed cio_close use when ENABLE_OSM_CONSOLE_SOCKET is not set In-Reply-To: <49B4EFCC.2050404@ext.bull.net> References: <49B4EFCC.2050404@ext.bull.net> Message-ID: <20090309144659.GK15398@sashak.voltaire.com> On 11:30 Mon 09 Mar , Nicolas Morey Chaisemartin wrote: > Fixes compilation problem introduced by 9b8561ac02e5e90c616b58bbbcfe2eb1a25d42fd > opensm/console: Fixed osm_console poll to handle POLLHUP > > Signed-off-by: Nicolas Morey-Chaisemartin Applied. Thanks. Sasha From sashak at voltaire.com Mon Mar 9 08:40:00 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Mar 2009 17:40:00 +0200 Subject: [ofa-general] Re: Question: remove c99 definitions within the ib_mad_f structure In-Reply-To: <49B4CFC6.7020800@gmail.com> References: <49B4CFC6.7020800@gmail.com> Message-ID: <20090309154000.GL15398@sashak.voltaire.com> Hi Eli, [adding ofw list...] On 10:13 Mon 09 Mar , Eli Dorfman (Voltaire) wrote: > > Can we revert commit 2dbb8b95d9dc27423a6fdb85d88ef385ecee0005 ? > "remove c99 definitions within the ib_mad_f structure > to enable better portability with WinOF." I suppose that just reverting will break WinOF. The whole discussion started somewhere at: http://lists.openfabrics.org/pipermail/general/2008-December/055902.html > Using the definitions makes the code clearer and safer to maintain. I agree and would prefer to have C99 "on" in our tree - it is really useful. > Aren't there better alternative, such as using gcc for example? WinOF Guys! This is one more call to find a way to coexist with C99. Sasha From sashak at voltaire.com Mon Mar 9 08:40:51 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Mar 2009 17:40:51 +0200 Subject: [ofa-general] Re: git head opensm build failure In-Reply-To: References: Message-ID: <20090309154051.GM15398@sashak.voltaire.com> Hi Or, On 12:10 Mon 09 Mar , Or Gerlitz wrote: > > Attempting to build the head of the mng git tree, I come > across the following error with opensm linking > > osm_console.o: In function `osm_console': opensm/opensm/osm_console.c:1471: > undefined reference to `cio_close' > > from the code it seems like a call to cio_close which is not under ifdef > ENABLE_OSM_CONSOLE_SOCKET Should be fixed now. Thanks for reporting. Sasha From ogerlitz at Voltaire.com Mon Mar 9 08:57:15 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Mon, 09 Mar 2009 17:57:15 +0200 Subject: [ofa-general] [PATCH v2 4/4] perfquery PortXmtDataSL/PortRcvDataSL support In-Reply-To: References: Message-ID: <49B53C5B.8070204@Voltaire.com> Or Gerlitz wrote: > perfquery PortXmtDataSL/PortRcvDataSL (IBA A13.6.5/6) support Hi Sasha, running with git head, I get an error on the per-SL Xmit/Rcv counters, I think it was introduced by one of the commits to the "new" API (e.g commit 17c6fb9787e5d3a10de97f555fcb0b130c4ede99 Author: Ira Weiny Date: Wed Feb 18 16:37:36 2009 -0800 libibmad: Clean up "new" interface BTW - I noted that a bunch of commits from Ira carry the date they were sent and not when there were applied... anyway, below is the error, and I will be sending now a patch that fixes it. Or. ibwarn: [25823] umad_send: fd -1 agentid 0 umad 0x7fff54fb1540 timeout 1000 ibwarn: [25823] umad_dump: agent id 0 status 0 timeout 1000 ibwarn: [25823] umad_addr_dump: qpn 1 qkey 0x80010000 lid 14 sl 0 grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x0 Gid 0x00000000000000000000000000000000 ibwarn: [25823] umad_send: write returned -1 != sizeof umad 64 + length 256 (Bad file descriptor) ibwarn: [25823] _do_madrpc: send failed; Bad file descriptor ibwarn: [25823] mad_rpc: _do_madrpc failed; dport (Lid 14) ./infiniband-diags/src/perfquery: iberror: [pid 25823] xmt_sl_query: failed: perfslquery From ogerlitz at voltaire.com Mon Mar 9 09:01:02 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Mon, 9 Mar 2009 18:01:02 +0200 (IST) Subject: [ofa-general] [PATCH] infiniband-diags/perfquery: add srcport param Message-ID: Add srcport param for the PortXmtDataSL/PortRcvDataSL calls made to libibmad Signed-off-by: Or Gerlitz Index: management/infiniband-diags/src/perfquery.c =================================================================== --- management.orig/infiniband-diags/src/perfquery.c +++ management/infiniband-diags/src/perfquery.c @@ -321,13 +321,13 @@ void xmt_sl_query(ib_portid_t *portid, i if (reset_only) { if (!performance_reset_via(pc, portid, port, mask, ibd_timeout, - IB_GSI_PORT_XMIT_DATA_SL, NULL)) + IB_GSI_PORT_XMIT_DATA_SL, srcport)) IBERROR("perfslreset"); return; } if (!pma_query_via(pc, portid, port, ibd_timeout, - IB_GSI_PORT_XMIT_DATA_SL, NULL)) + IB_GSI_PORT_XMIT_DATA_SL, srcport)) IBERROR("perfslquery"); mad_dump_perfcounters_xmt_sl(buf, sizeof buf, pc, sizeof pc); @@ -335,7 +335,7 @@ void xmt_sl_query(ib_portid_t *portid, i if(reset) if (!performance_reset_via(pc, portid, port, mask, ibd_timeout, - IB_GSI_PORT_XMIT_DATA_SL, NULL)) + IB_GSI_PORT_XMIT_DATA_SL, srcport)) IBERROR("perfslreset"); } @@ -345,13 +345,13 @@ void rcv_sl_query(ib_portid_t *portid, i if (reset_only) { if (!performance_reset_via(pc, portid, port, mask, ibd_timeout, - IB_GSI_PORT_RCV_DATA_SL, NULL)) + IB_GSI_PORT_RCV_DATA_SL, srcport)) IBERROR("perfslreset"); return; } if (!pma_query_via(pc, portid, port, ibd_timeout, - IB_GSI_PORT_RCV_DATA_SL, NULL)) + IB_GSI_PORT_RCV_DATA_SL, srcport)) IBERROR("perfslquery"); mad_dump_perfcounters_rcv_sl(buf, sizeof buf, pc, sizeof pc); @@ -359,7 +359,7 @@ void rcv_sl_query(ib_portid_t *portid, i if(reset) if (!performance_reset_via(pc, portid, port, mask, ibd_timeout, - IB_GSI_PORT_RCV_DATA_SL, NULL)) + IB_GSI_PORT_RCV_DATA_SL, srcport)) IBERROR("perfslreset"); } From sashak at voltaire.com Mon Mar 9 10:42:18 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Mar 2009 19:42:18 +0200 Subject: [ofa-general] Re: [PATCH] infiniband-diags/perfquery: add srcport param In-Reply-To: References: Message-ID: <20090309174218.GN15398@sashak.voltaire.com> On 18:01 Mon 09 Mar , Or Gerlitz wrote: > Add srcport param for the PortXmtDataSL/PortRcvDataSL calls made to libibmad > > Signed-off-by: Or Gerlitz Makes sense. Applied. Thanks. Sasha From sashak at voltaire.com Mon Mar 9 10:46:43 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 9 Mar 2009 19:46:43 +0200 Subject: [ofa-general] [PATCH v2 4/4] perfquery PortXmtDataSL/PortRcvDataSL support In-Reply-To: <49B53C5B.8070204@Voltaire.com> References: <49B53C5B.8070204@Voltaire.com> Message-ID: <20090309174625.GO15398@sashak.voltaire.com> On 17:57 Mon 09 Mar , Or Gerlitz wrote: > > running with git head, I get an error on the per-SL Xmit/Rcv counters, I think it was introduced by one of the commits to the "new" API (e.g It was resulted by my rebase, not by patches. > > commit 17c6fb9787e5d3a10de97f555fcb0b130c4ede99 > Author: Ira Weiny > Date: Wed Feb 18 16:37:36 2009 -0800 > > libibmad: Clean up "new" interface > > BTW - I noted that a bunch of commits from Ira carry the date they were sent and not when there were applied... Yes, it is normal - you are looking on 'AuthorDate', not 'CommitDate'. Try 'git log --pretty=fuller'. > anyway, below is the error, and I will be sending now a patch that fixes it. Thanks. Sasha From sean.hefty at intel.com Mon Mar 9 11:15:17 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 9 Mar 2009 11:15:17 -0700 Subject: [ofa-general] RE: Question: remove c99 definitions within the ib_mad_f structure In-Reply-To: <20090309154000.GL15398@sashak.voltaire.com> References: <49B4CFC6.7020800@gmail.com> <20090309154000.GL15398@sashak.voltaire.com> Message-ID: <8436BA8017D247AC9287E3F0DB4458DB@amr.corp.intel.com> >I suppose that just reverting will break WinOF. The whole discussion >started somewhere at: This is getting old and frustrating. Do the OFA IB vendors not want to support Windows based systems connected to their hardware? From swise at opengridcomputing.com Mon Mar 9 11:17:57 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 09 Mar 2009 13:17:57 -0500 Subject: [ofa-general] Re: tcp/rdma port unification patch in librdmacm In-Reply-To: <811553D6D34A4E6F9AB686CA2E441D35@amr.corp.intel.com> References: <49B17E15.4050406@opengridcomputing.com> <811553D6D34A4E6F9AB686CA2E441D35@amr.corp.intel.com> Message-ID: <49B55D55.4090005@opengridcomputing.com> Sean Hefty wrote: >> Would you reconsider moving the tcp/rdma port unification changes up >> into librdmacm? The benefit is that this will resolve the issue in a >> manner that can flow into the distros and not affect the linux kernel. >> So librdmacm could, if configured to support this, just allocate and >> bind a user socket to reserve the TCP port on behalf of the rdma >> application. The code would be similar to the kernel patch... >> >> Thoughts? >> > > This is what I ended up needing to do for Windows (but for other reasons), so > I'm not against adding this to the library. > > I'm undecided on the best way to expose this. Always enabled? Enabled during > the build? Enabled through a new API? Other? > > - Sean > Shall I post a patch for comments? From kliteyn at dev.mellanox.co.il Mon Mar 9 11:26:12 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Mon, 09 Mar 2009 20:26:12 +0200 Subject: [ofa-general] Re: opensm compilation fails In-Reply-To: <20090309142914.GJ15398@sashak.voltaire.com> References: <49B4E398.6050701@gmail.com> <49B4E533.4030208@ext.bull.net> <20090309135826.GF15398@sashak.voltaire.com> <49B5202B.5040706@dev.mellanox.co.il> <20090309142914.GJ15398@sashak.voltaire.com> Message-ID: <49B55F44.7040004@dev.mellanox.co.il> Sasha Khapyorsky wrote: > Yevgeny, > > BTW, could we do something with this warning (gcc-4.3.3)? > > osm_qos_parser_l.c: In function 'yylex': > osm_qos_parser_l.l:358: warning: ignoring return value of 'fwrite', declared with attribute warn_unused_result Didn't see this warning (using gcc-4.1.0). I'll take care of it. -- Yevgeny > Sasha > From weiny2 at llnl.gov Mon Mar 9 11:32:33 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Mon, 9 Mar 2009 11:32:33 -0700 Subject: [ofa-general] RE: Question: remove c99 definitions within the ib_mad_f structure In-Reply-To: <8436BA8017D247AC9287E3F0DB4458DB@amr.corp.intel.com> References: <49B4CFC6.7020800@gmail.com> <20090309154000.GL15398@sashak.voltaire.com> <8436BA8017D247AC9287E3F0DB4458DB@amr.corp.intel.com> Message-ID: <20090309113233.b382dc70.weiny2@llnl.gov> On Mon, 9 Mar 2009 11:15:17 -0700 "Sean Hefty" wrote: > >I suppose that just reverting will break WinOF. The whole discussion > >started somewhere at: > > This is getting old and frustrating. Do the OFA IB vendors not want to support > Windows based systems connected to their hardware? > From weiny2 at llnl.gov Mon Mar 9 11:35:30 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Mon, 9 Mar 2009 11:35:30 -0700 Subject: [ofa-general] RE: Question: remove c99 definitions within the ib_mad_f structure In-Reply-To: <20090309113233.b382dc70.weiny2@llnl.gov> References: <49B4CFC6.7020800@gmail.com> <20090309154000.GL15398@sashak.voltaire.com> <8436BA8017D247AC9287E3F0DB4458DB@amr.corp.intel.com> <20090309113233.b382dc70.weiny2@llnl.gov> Message-ID: <20090309113530.8b056000.weiny2@llnl.gov> Oopps clicked send a bit too early. On Mon, 9 Mar 2009 11:32:33 -0700 Ira Weiny wrote: > On Mon, 9 Mar 2009 11:15:17 -0700 > "Sean Hefty" wrote: > > > >I suppose that just reverting will break WinOF. The whole discussion > > >started somewhere at: > > > > This is getting old and frustrating. Do the OFA IB vendors not want to support > > Windows based systems connected to their hardware? > > > I have no problem with it but I don't have a Windows machine to test on. So this will make the effort harder on your part Sean, sorry. Ira -- Ira Weiny Math Programer/Computer Scientist Larence Livermore National Lab weiny2 at llnl.gov From weiny2 at llnl.gov Mon Mar 9 11:39:18 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Mon, 9 Mar 2009 11:39:18 -0700 Subject: [ofa-general] ***SPAM*** problem about the OpenSM In-Reply-To: <91fe68d50903090441t114d70abv2f5091d00d3d185b@mail.gmail.com> References: <91fe68d50903090441t114d70abv2f5091d00d3d185b@mail.gmail.com> Message-ID: <20090309113918.167a1b10.weiny2@llnl.gov> On Mon, 9 Mar 2009 19:41:05 +0800 Jordan wrote: > I want to do some research about the routing algorithm in the OpenSM, for > example, adding a new routing algorithm (e.g for torus or mesh)or improving > some of them. > > But, after re-configuring and rebuilding the OpenSM, can the OpenSM still > be installed on Mellanox switch? I don't know the answer to this as I don't know how OpenSM is installed, if at all, on a Mellanox Switch. > > Can the new routing algorithm take effect in the network? What steps can I > take to achieve this purpose? I would start by turning off the SM on the Mellanox Switch and running OpenSM on one of your nodes. At that point OpenSM will be routing the fabric. If you can't disable the SM on the switch you could try running OpenSM at a higher priority. To help determine which SM is master the OpenSM console now has additional SM information in the "status" command which will report all SM's in the system as well as which one is master. Hope this helps, Ira -- Ira Weiny Math Programer/Computer Scientist Larence Livermore National Lab weiny2 at llnl.gov From bart.vanassche at gmail.com Mon Mar 9 11:47:39 2009 From: bart.vanassche at gmail.com (Bart Van Assche) Date: Mon, 9 Mar 2009 19:47:39 +0100 Subject: [ofa-general] ***SPAM*** InfiniBand and PCIe 2.0 Message-ID: Hello, Although I'm not entirely sure this is the right mailing list for such questions: can anyone give me some advice on how to get an InfiniBand HCA working at 5 GT/s ? I have inserted a MT25418 HCA in a PCIe 2.0 slot. According to Mellanox' documentation, the maximum transfer speed should be 5.0 GT/s. lspci reports 2.5 GT/s for the link status however. The details are as follows: * MT25418 specs: http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=36&menu_section=34 * Kernel: Linux 2.6.28.7 * Motherboard: Asus P5Q Deluxe * lspci output: 01:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0) Subsystem: Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Capabilities: [9c] MSI-X: Enable+ Mask- TabSize=256 Vector table: BAR=4 offset=00000000 PBA: BAR=4 offset=00001000 Capabilities: [60] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited ClockPM- Suprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB Kernel driver in use: mlx4_core Kernel modules: mlx4_core Bart. -------------- next part -------------- An HTML attachment was scrubbed... URL: From boris at mellanox.com Mon Mar 9 11:54:03 2009 From: boris at mellanox.com (Boris Shpolyansky) Date: Mon, 9 Mar 2009 11:54:03 -0700 Subject: [ofa-general] ***SPAM*** InfiniBand and PCIe 2.0 References: Message-ID: <1E3DCD1C63492545881FACB6063A57C1038A7FD5@mtiexch01> Bart, Particular card you were testing doesn't support 5 GT/s PCIe operation. There are other adaptors from the same product family based on ConnectX silicon device that are capable of PCIe gen-2 (5GT/s) link speed. Please, contact your distributor for details. Boris Shpolyansky Sr. Member of Technical Staff, Applications Mellanox Technologies Inc. 350 Oakmead Parkway, Suite 100 Sunnyvale, CA 94085 Tel.: (408) 916 0014 Fax: (408) 585 0314 Cell: (408) 834 9365 www.mellanox.com ________________________________ From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Bart Van Assche Sent: Monday, March 09, 2009 11:48 AM To: general at lists.openfabrics.org Subject: [ofa-general] ***SPAM*** InfiniBand and PCIe 2.0 Hello, Although I'm not entirely sure this is the right mailing list for such questions: can anyone give me some advice on how to get an InfiniBand HCA working at 5 GT/s ? I have inserted a MT25418 HCA in a PCIe 2.0 slot. According to Mellanox' documentation, the maximum transfer speed should be 5.0 GT/s. lspci reports 2.5 GT/s for the link status however. The details are as follows: * MT25418 specs: http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family =36&menu_section=34 * Kernel: Linux 2.6.28.7 * Motherboard: Asus P5Q Deluxe * lspci output: 01:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0) Subsystem: Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Capabilities: [9c] MSI-X: Enable+ Mask- TabSize=256 Vector table: BAR=4 offset=00000000 PBA: BAR=4 offset=00001000 Capabilities: [60] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #8, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited ClockPM- Suprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB Kernel driver in use: mlx4_core Kernel modules: mlx4_core Bart. -------------- next part -------------- An HTML attachment was scrubbed... URL: From weiny2 at llnl.gov Mon Mar 9 14:26:08 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Mon, 9 Mar 2009 14:26:08 -0700 Subject: [ofa-general] Re: Fwd: [PATCH] infiniband-diags/ibnetdiscover: Apply --node-name-map remapping to names printed by --ports option In-Reply-To: <5e3be0f30903071931q2dc20be5mc2403d6fd1e62011@mail.gmail.com> References: <49B32AAE.mailMM1C6FUZ@systemfabricworks.com> <5e3be0f30903071931q2dc20be5mc2403d6fd1e62011@mail.gmail.com> Message-ID: <20090309142608.5b087caf.weiny2@llnl.gov> Sasha, David sent this to me to put the node names in the output of the ports option. It did not apply; I guess because of mailer issues. So I applied by hand and created the attached patch with David's signed off by line. The first chunk did not seem to be more than a format change so I left it out. Was there something there David? Ira On Sat, 7 Mar 2009 21:31:38 -0600 David McMillen wrote: > I sent a patch to the general mailing list, but my "message to general > awaits moderator approval" > > Since you seem to be working the tree a lot, I thought I'd pass it on to you > as a heads-up. > > I hope this mailer didn't mess up the tabs/spaces... I know the one sent to > the list was done right. > > Dave > > > ---------- Forwarded message ---------- > From: > Date: Sat, Mar 7, 2009 at 8:17 PM > Subject: [PATCH] infiniband-diags/ibnetdiscover: Apply --node-name-map > remapping to names printed by --ports option > To: general at lists.openfabrics.org, sashak at voltaire.com > > > > Apply --node-name-map remapping to names printed by --ports option. > > Signed-off-by: David A. McMillen > --- > infiniband-diags/src/ibnetdiscover.c | 17 +++++++++++++---- > 1 files changed, 13 insertions(+), 4 deletions(-) > > diff --git a/infiniband-diags/src/ibnetdiscover.c > b/infiniband-diags/src/ibnetdiscover.c > index 948a79d..3ace235 100644 > --- a/infiniband-diags/src/ibnetdiscover.c > +++ b/infiniband-diags/src/ibnetdiscover.c > @@ -903,15 +903,24 @@ void dump_ports_report () > fprintf(stdout, > " - %2s %5d %2d 0x%016" > PRIx64 > " ( '%s' - '%s' )\n", > - > node_type_str2(port->remoteport->node), > + node_type_str2(port-> > + remoteport->node), > port->remoteport->lid, > port->remoteport->portnum, > port->remoteport->portguid, > - port->node->nodedesc, > - > port->remoteport->node->nodedesc); > + > remap_node_name(node_name_map, > + port->node->nodeguid, > + port->node->nodedesc), > + > remap_node_name(node_name_map, > + port->remoteport->node-> > + nodeguid, > + port->remoteport->node-> > + nodedesc)); > else > fprintf(stdout, "%36s'%s'\n", "", > - port->node->nodedesc); > + > remap_node_name(node_name_map, > + port->node->nodeguid, > + port->node->nodedesc)); > } > n++; > } > -- Ira Weiny Math Programer/Computer Scientist Larence Livermore National Lab weiny2 at llnl.gov -------------- next part -------------- A non-text attachment was scrubbed... Name: 0001-infiniband-diags-ibnetdiscover-Apply-node-name-ma.patch Type: application/octet-stream Size: 1342 bytes Desc: not available URL: From sean.hefty at intel.com Mon Mar 9 14:36:20 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 9 Mar 2009 14:36:20 -0700 Subject: [ofa-general] RE: tcp/rdma port unification patch in librdmacm In-Reply-To: <49B55D55.4090005@opengridcomputing.com> References: <49B17E15.4050406@opengridcomputing.com> <811553D6D34A4E6F9AB686CA2E441D35@amr.corp.intel.com> <49B55D55.4090005@opengridcomputing.com> Message-ID: >Shall I post a patch for comments? I think that would be the best way forward. I've been trying to run through different use case scenarios in my head to see if there is any security or other issue with this approach. From the kernel's perspective, it seems like this adds some sort of trust that an application has reserved a desired port number from the host stack before attempting to use it on the RDMA side. I'm just not sure if this leads to some real problem or not. - Sean From ben at lanl.gov Mon Mar 9 16:19:13 2009 From: ben at lanl.gov (Benjamin M. McClelland) Date: Mon, 9 Mar 2009 17:19:13 -0600 (MDT) Subject: [ofa-general] ipoib performanec at smaller mtu Message-ID: <59748.128.165.248.31.1236640753.squirrel@webmail.lanl.gov> I was curious what other people are seeing for 1500 mtu performance on ipoib. Current tests are showing that I get about 150-175MB/s no matter how many streams. If I go up to higher mtu, then performance significantly increases. Due to the storage network that this ib is routing to, we are currently limited to 1500 mtu. ofed 1.4.1, connected mode, dual opteron dual core 1.8GHz, Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0) fw 2.6.000, voltaire sm on switch lro disabled, send_queue_size=256 thanks, -Ben From sashak at voltaire.com Mon Mar 9 17:06:04 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Mar 2009 02:06:04 +0200 Subject: [ofa-general] Re: Question: remove c99 definitions within the ib_mad_f structure In-Reply-To: <8436BA8017D247AC9287E3F0DB4458DB@amr.corp.intel.com> References: <49B4CFC6.7020800@gmail.com> <20090309154000.GL15398@sashak.voltaire.com> <8436BA8017D247AC9287E3F0DB4458DB@amr.corp.intel.com> Message-ID: <20090310000604.GA10521@sashak.voltaire.com> On 11:15 Mon 09 Mar , Sean Hefty wrote: > > This is getting old and frustrating. Do the OFA IB vendors not want to support > Windows based systems connected to their hardware? I don't think that it about supporting Windows or not, rather it is about having a better code. For both - linux and WinOF. Sasha From boris at mellanox.com Mon Mar 9 17:09:34 2009 From: boris at mellanox.com (Boris Shpolyansky) Date: Mon, 9 Mar 2009 17:09:34 -0700 Subject: [ofa-general] ipoib performanec at smaller mtu References: <59748.128.165.248.31.1236640753.squirrel@webmail.lanl.gov> Message-ID: <1E3DCD1C63492545881FACB6063A57C1038A80F5@mtiexch01> I think for small MTU sizes you will see better performance with IPoIB in datagram mode. Boris Shpolyansky Sr. Member of Technical Staff, Applications Mellanox Technologies Inc. 350 Oakmead Parkway, Suite 100 Sunnyvale, CA 94085 Tel.: (408) 916 0014 Fax: (408) 585 0314 Cell: (408) 834 9365 www.mellanox.com -----Original Message----- From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Benjamin M. McClelland Sent: Monday, March 09, 2009 4:19 PM To: general at lists.openfabrics.org Subject: [ofa-general] ipoib performanec at smaller mtu I was curious what other people are seeing for 1500 mtu performance on ipoib. Current tests are showing that I get about 150-175MB/s no matter how many streams. If I go up to higher mtu, then performance significantly increases. Due to the storage network that this ib is routing to, we are currently limited to 1500 mtu. ofed 1.4.1, connected mode, dual opteron dual core 1.8GHz, Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0) fw 2.6.000, voltaire sm on switch lro disabled, send_queue_size=256 thanks, -Ben _______________________________________________ general mailing list general at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From ben at lanl.gov Mon Mar 9 17:19:23 2009 From: ben at lanl.gov (Ben) Date: Mon, 9 Mar 2009 18:19:23 -0600 Subject: [ofa-general] ipoib performanec at smaller mtu In-Reply-To: <1E3DCD1C63492545881FACB6063A57C1038A80F5@mtiexch01> References: <59748.128.165.248.31.1236640753.squirrel@webmail.lanl.gov> <1E3DCD1C63492545881FACB6063A57C1038A80F5@mtiexch01> Message-ID: <03EC693D-38BE-4947-8D7A-7C8CC3255815@lanl.gov> I have a similar setup in datagram, and don't see a big difference. Are these numbers expected? I guess I would like to understand the limitations. It seams like these numbers are a bit below what the hardware is capable of even with lro off. -Ben On Mar 9, 2009, at 6:09 PM, Boris Shpolyansky wrote: > I think for small MTU sizes you will see better performance with IPoIB > in datagram mode. > > Boris Shpolyansky > Sr. Member of Technical Staff, Applications > > Mellanox Technologies Inc. > 350 Oakmead Parkway, Suite 100 > Sunnyvale, CA 94085 > Tel.: (408) 916 0014 > Fax: (408) 585 0314 > Cell: (408) 834 9365 > www.mellanox.com > > -----Original Message----- > From: general-bounces at lists.openfabrics.org > [mailto:general-bounces at lists.openfabrics.org] On Behalf Of > Benjamin M. > McClelland > Sent: Monday, March 09, 2009 4:19 PM > To: general at lists.openfabrics.org > Subject: [ofa-general] ipoib performanec at smaller mtu > > I was curious what other people are seeing for 1500 mtu performance on > ipoib. Current tests are showing that I get about 150-175MB/s no > matter > how many streams. If I go up to higher mtu, then performance > significantly increases. Due to the storage network that this ib is > routing to, we are currently limited to 1500 mtu. > > ofed 1.4.1, connected mode, dual opteron dual core 1.8GHz, Mellanox > Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0) fw > 2.6.000, voltaire sm on switch > > lro disabled, send_queue_size=256 > > thanks, > -Ben > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general From andy.grover at oracle.com Mon Mar 9 18:38:01 2009 From: andy.grover at oracle.com (Andy Grover) Date: Mon, 09 Mar 2009 18:38:01 -0700 Subject: [ofa-general] RDD and EE context support Message-ID: <49B5C479.8060800@oracle.com> I've been grepping and Googling around about Reliable Datagram Domains and End-to-End Contexts. These features are in the IB 1.2.1 spec but appear to be entirely unimplemented. Is this the case? Just wanted to double-check before I gave up on them. Thanks -- Regards -- Andy From rdreier at cisco.com Mon Mar 9 20:03:06 2009 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 09 Mar 2009 20:03:06 -0700 Subject: [ofa-general] RDD and EE context support In-Reply-To: <49B5C479.8060800@oracle.com> (Andy Grover's message of "Mon, 09 Mar 2009 18:38:01 -0700") References: <49B5C479.8060800@oracle.com> Message-ID: > I've been grepping and Googling around about Reliable Datagram Domains > and End-to-End Contexts. These features are in the IB 1.2.1 spec but > appear to be entirely unimplemented. Is this the case? Just wanted to > double-check before I gave up on them. Correct... no hardware ever implemented the IB RD transport. - R. From ogerlitz at Voltaire.com Tue Mar 10 00:53:01 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Tue, 10 Mar 2009 09:53:01 +0200 Subject: [ofa-general] RE: tcp/rdma port unification patch in librdmacm In-Reply-To: <811553D6D34A4E6F9AB686CA2E441D35@amr.corp.intel.com> References: <49B17E15.4050406@opengridcomputing.com> <811553D6D34A4E6F9AB686CA2E441D35@amr.corp.intel.com> Message-ID: <49B61C5D.6060900@Voltaire.com> Sean Hefty wrote: > I'm undecided on the best way to expose this. Always enabled? Enabled during > the build? Enabled through a new API? Other? Hi Sean, Experimenting with the kernel patch (below) applied on Linus tree, such that the rdma_cm module is loaded with the param being ON and running rping, I didn't see any extra socket listed in the netstat -natp output. So to start with (I tend to think it was explained in the past... sorry for asking again) - is it correct that the approach taken by this patch is to consume only a --port-- from the tcp port space but not to have actually a visible socket? is this approach applicable to user space? As for enabling this always on librdmacm, this can't work - since it will break IB user space app that attempts to open two listeners: TCP one on port X and IB one on SID(PS_TCP, X) - for example for X=3260 and the app being the Linux stgt target. Also Steve has mentioned earlier on this thread that kernel apps are safe from this problem - I am not sure this is the case for Lustre. Or. RDMA/CMA: Allocate PS_TCP ports from the host TCP port space. This is needed for iwarp providers that support native and rdma connections over the same interface. Optionally turned on by the new rdma_cm unify_tcp_port_space module option. Signed-off-by: Steve Wise Signed-off-by: Faisal Latif Signed-off-by: Chien Tung --- drivers/infiniband/core/cma.c | 43 ++++++++++++++++++++++++++++++++++++++++- 1 files changed, 42 insertions(+), 1 deletions(-) Index: linus-linux-2.6/drivers/infiniband/core/cma.c =================================================================== --- linus-linux-2.6.orig/drivers/infiniband/core/cma.c +++ linus-linux-2.6/drivers/infiniband/core/cma.c @@ -55,6 +55,11 @@ MODULE_AUTHOR("Sean Hefty"); MODULE_DESCRIPTION("Generic RDMA CM Agent"); MODULE_LICENSE("Dual BSD/GPL"); +int unify_tcp_port_space = 0; +module_param(unify_tcp_port_space, int, 0644); +MODULE_PARM_DESC(unify_tcp_port_space, "Unify the host TCP and RDMA port " + "space allocation (default=0)"); + #define CMA_CM_RESPONSE_TIMEOUT 20 #define CMA_MAX_CM_RETRIES 15 #define CMA_CM_MRA_SETTING (IB_CM_MRA_FLAG_DELAY | 24) @@ -118,6 +123,7 @@ struct rdma_id_private { struct rdma_cm_id id; struct rdma_bind_list *bind_list; + struct socket *sock; struct hlist_node node; struct list_head list; /* listen_any_list or cma_device.list */ struct list_head listen_list; /* per device listens */ @@ -813,6 +819,8 @@ static void cma_release_port(struct rdma kfree(bind_list); } mutex_unlock(&lock); + if (id_priv->sock) + sock_release(id_priv->sock); } static void cma_leave_mc_groups(struct rdma_id_private *id_priv) @@ -2042,6 +2050,34 @@ static int cma_use_port(struct idr *ps, return 0; } +static int cma_get_tcp_port(struct rdma_id_private *id_priv) +{ + int ret; + int size; + struct socket *sock; + + ret = sock_create_kern(AF_INET, SOCK_STREAM, IPPROTO_TCP, &sock); + if (ret) + return ret; + ret = sock->ops->bind(sock, + (struct sockaddr *) &id_priv->id.route.addr.src_addr, + ip_addr_size((struct sockaddr *) &id_priv->id.route.addr.src_addr)); + if (ret) { + sock_release(sock); + return ret; + } + size = ip_addr_size(&id_priv->id.route.addr.src_addr); + ret = sock->ops->getname(sock, + (struct sockaddr *) &id_priv->id.route.addr.src_addr, + &size, 0); + if (ret) { + sock_release(sock); + return ret; + } + id_priv->sock = sock; + return 0; +} + static int cma_get_port(struct rdma_id_private *id_priv) { struct idr *ps; @@ -2053,6 +2089,11 @@ static int cma_get_port(struct rdma_id_p break; case RDMA_PS_TCP: ps = &tcp_ps; + if (unify_tcp_port_space) { + ret = cma_get_tcp_port(id_priv); + if (ret) + goto out; + } break; case RDMA_PS_UDP: ps = &udp_ps; @@ -2070,7 +2111,7 @@ static int cma_get_port(struct rdma_id_p else ret = cma_use_port(ps, id_priv); mutex_unlock(&lock); - +out: return ret; } From sashak at voltaire.com Tue Mar 10 02:17:56 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Mar 2009 11:17:56 +0200 Subject: [ofa-general] Re: Fwd: [PATCH] infiniband-diags/ibnetdiscover: Apply --node-name-map remapping to names printed by --ports option In-Reply-To: <20090309142608.5b087caf.weiny2@llnl.gov> References: <49B32AAE.mailMM1C6FUZ@systemfabricworks.com> <5e3be0f30903071931q2dc20be5mc2403d6fd1e62011@mail.gmail.com> <20090309142608.5b087caf.weiny2@llnl.gov> Message-ID: <20090310091756.GA23898@sashak.voltaire.com> On 14:26 Mon 09 Mar , Ira Weiny wrote: > Sasha, David sent this to me to put the node names in the output of the ports > option. It did not apply; I guess because of mailer issues. This applied fine for me (using 'git am'). > So I applied by hand and created the attached patch with David's signed off by > line. Applied. Thanks. Sasha From sashak at voltaire.com Tue Mar 10 02:18:16 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Mar 2009 11:18:16 +0200 Subject: [ofa-general] Re: [PATCH] opensm/PerfMgr: A few more esp0 changes In-Reply-To: <20090306152554.GA29312@comcast.net> References: <20090306152554.GA29312@comcast.net> Message-ID: <20090310091816.GB23898@sashak.voltaire.com> On 10:25 Fri 06 Mar , Hal Rosenstock wrote: > > Also, some cosmetic changes > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Tue Mar 10 02:28:46 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Mar 2009 11:28:46 +0200 Subject: [ofa-general] Re: [PATCH 1/2] opensm: Handle trap repress on trap 144 generation In-Reply-To: References: <20090305122433.GB20341@comcast.net> <20090306173026.GA9673@sashak.voltaire.com> <20090306204123.GG9673@sashak.voltaire.com> Message-ID: <20090310092846.GC23898@sashak.voltaire.com> Hi Hal, On 16:12 Fri 06 Mar , Hal Rosenstock wrote: > Maybe I didn't misread. > > > trap_rcv needs to cause the repress for the trap to be generated. > > This is already done. > > I think what you were saying was why does the incoming repress need to > get this far into the code, right ? Not really. I don't see how TrapRepress can reach trap_rcv_process_response() function. I will comment over v2 of the patch. Sasha From sashak at voltaire.com Tue Mar 10 02:31:54 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Mar 2009 11:31:54 +0200 Subject: [ofa-general] Re: [PATCH v2 1/2] opensm: Handle trap repress on trap 144 generation In-Reply-To: <20090306181948.GA1706@comcast.net> References: <20090306181948.GA1706@comcast.net> Message-ID: <20090310093154.GD23898@sashak.voltaire.com> On 13:19 Fri 06 Mar , Hal Rosenstock wrote: > diff --git a/opensm/opensm/osm_trap_rcv.c b/opensm/opensm/osm_trap_rcv.c > index 6639246..4576a8b 100644 > --- a/opensm/opensm/osm_trap_rcv.c > +++ b/opensm/opensm/osm_trap_rcv.c > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2009 HNR Consulting. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -649,18 +650,34 @@ Exit: > } > > /********************************************************************** > - CURRENTLY WE ARE NOT CREATING TRAPS - SO THIS CALL IN AN ERROR > **********************************************************************/ > static void > trap_rcv_process_response(IN osm_sm_t * sm, > IN const osm_madw_t * const p_madw) > { > + uint8_t payload[sizeof(ib_mad_notice_attr_t)]; > + ib_smp_t *p_smp; > + ib_mad_notice_attr_t *p_ntci = (ib_mad_notice_attr_t *) payload; > + > + CL_ASSERT(p_madw); > + > + if (!osm_log_is_active(sm->p_log, OSM_LOG_VERBOSE)) > + return; > > OSM_LOG_ENTER(sm->p_log); > > - OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3808: " > - "This function is not supported yet\n"); > + if (p_madw->p_mad->mgmt_class != IB_MCLASS_SUBN_LID && > + p_madw->p_mad->mgmt_class != IB_MCLASS_SUBN_DIR) > + goto Exit; > > + p_smp = osm_madw_get_smp_ptr(p_madw); > + > + memset(payload, 0, sizeof(payload)); > + memcpy(payload, &p_smp->data, IB_SMP_DATA_SIZE); > + > + osm_dump_notice(sm->p_log, p_ntci, OSM_LOG_VERBOSE); > + > +Exit: > OSM_LOG_EXIT(sm->p_log); > } Almost fine, but one thing still be unclear for me. How is this function trap_rcv_process_response() reachable and what is this supposed to do? Sasha From Line.Holen at Sun.COM Tue Mar 10 02:37:39 2009 From: Line.Holen at Sun.COM (Line.Holen at Sun.COM) Date: Tue, 10 Mar 2009 10:37:39 +0100 Subject: [ofa-general] [PATCH v2] opensm/osm_link_mgr.c initialize SMSL Message-ID: <49B634E3.7080006@Sun.COM> The SMSL is currently not initialized. Whatever is written in PortInfo at power up is kept. The lash routing algorithm makes active use of SL to avoid deadlock in the fabric. The endnodes needs to be instructed to use the SL calculated for the endnode-to-SMnode path for SA traffic. If not, the SA traffic can cause deadlock. This change makes sure that the SMSL is properly initialized on all ports. If flash routing is used, the SMSL is set up based on routing calculations. For other routing algorithms the default SMSL is used. Signed-off-by: Line Holen --- diff --git a/opensm/include/opensm/osm_ucast_lash.h b/opensm/include/opensm/osm_ucast_lash.h index c1397e2..3416e02 100644 --- a/opensm/include/opensm/osm_ucast_lash.h +++ b/opensm/include/opensm/osm_ucast_lash.h @@ -44,6 +44,8 @@ #ifndef OSM_UCAST_LASH_H #define OSM_UCAST_LASH_H +#include + enum { UNQUEUED, Q_MEMBER, @@ -92,4 +94,9 @@ typedef struct _lash { int ***virtual_location; } lash_t; +uint8_t +osm_get_lash_sl(osm_opensm_t * p_osm, + const osm_port_t * p_src_port, + const osm_port_t * p_dst_port); + #endif diff --git a/opensm/opensm/osm_link_mgr.c b/opensm/opensm/osm_link_mgr.c index 37e3e1b..c88da29 100644 --- a/opensm/opensm/osm_link_mgr.c +++ b/opensm/opensm/osm_link_mgr.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -51,6 +52,44 @@ #include #include #include +#include +#include + + +/********************************************************************** + **********************************************************************/ +static uint8_t +link_mgr_get_smsl(IN osm_sm_t * sm, IN osm_physp_t * const p_physp) +{ + osm_opensm_t *p_osm = sm->p_subn->p_osm; + const osm_port_t *p_sm_port; + const osm_port_t *p_src_port; + ib_net16_t slid; + ib_net16_t smlid; + uint8_t sl; + + OSM_LOG_ENTER(sm->p_log); + + if (p_osm->routing_engine_used != OSM_ROUTING_ENGINE_TYPE_LASH) { + /* Use default SL if lash routing is not used */ + OSM_LOG_EXIT(sm->p_log); + return (OSM_DEFAULT_SL); + } + + /* Find osm_port of the SM itself = dest_port */ + smlid = sm->p_subn->sm_base_lid; + p_sm_port = cl_ptr_vector_get(&sm->p_subn->port_lid_tbl, cl_ntoh16(smlid)); + + /* Find osm_port of the source = p_physp */ + slid = osm_physp_get_base_lid(p_physp); + p_src_port = cl_ptr_vector_get(&sm->p_subn->port_lid_tbl, cl_ntoh16(slid)); + + /* Call lash to find proper SL */ + sl = osm_get_lash_sl(p_osm, p_src_port, p_sm_port); + + OSM_LOG_EXIT(sm->p_log); + return (sl); +} /********************************************************************** **********************************************************************/ @@ -67,6 +106,7 @@ __osm_link_mgr_set_physp_pi(osm_sm_t * sm, ib_api_status_t status; uint8_t port_num; uint8_t mtu; + uint8_t smsl = OSM_DEFAULT_SL; uint8_t op_vls; boolean_t esp0 = FALSE; boolean_t send_set = FALSE; @@ -76,6 +116,8 @@ __osm_link_mgr_set_physp_pi(osm_sm_t * sm, p_node = osm_physp_get_node_ptr(p_physp); + p_old_pi = &p_physp->port_info; + port_num = osm_physp_get_port_num(p_physp); if (port_num == 0) { @@ -93,22 +135,30 @@ __osm_link_mgr_set_physp_pi(osm_sm_t * sm, if (ib_switch_info_is_enhanced_port0(&p_node->sw->switch_info) == FALSE) { - /* This means the switch doesn't support enhanced port 0. - Can skip it. */ - OSM_LOG(sm->p_log, OSM_LOG_DEBUG, - "Skipping port 0, GUID 0x%016" PRIx64 "\n", - cl_ntoh64(osm_physp_get_port_guid(p_physp))); - goto Exit; + + /* Even for base port 0 we might have to set smsl + (if we are using lash routing) */ + smsl = link_mgr_get_smsl(sm, p_physp); + if (smsl != ib_port_info_get_master_smsl(p_old_pi)) { + send_set = TRUE; + OSM_LOG(sm->p_log, OSM_LOG_DEBUG, + "Setting SMSL to %d on port 0 GUID 0x%016" + PRIx64 "\n", smsl, + cl_ntoh64(osm_physp_get_port_guid(p_physp))); + } else { + /* This means the switch doesn't support + enhanced port 0 and we don't need to + change SMSL. Can skip it. */ + OSM_LOG(sm->p_log, OSM_LOG_DEBUG, + "Skipping port 0, GUID 0x%016" PRIx64 "\n", + cl_ntoh64(osm_physp_get_port_guid(p_physp))); + goto Exit; + } + } else { + esp0 = TRUE; } - esp0 = TRUE; } - /* - PAST THIS POINT WE ARE HANDLING EITHER A NON PORT 0 OR ENHANCED PORT 0 - */ - - p_old_pi = &p_physp->port_info; - memset(payload, 0, IB_SMP_DATA_SIZE); memcpy(payload, p_old_pi, sizeof(ib_port_info_t)); @@ -124,6 +174,16 @@ __osm_link_mgr_set_physp_pi(osm_sm_t * sm, p_pi->state_info2 = 0x02; ib_port_info_set_port_state(p_pi, port_state); + /* Check whether this is base port0 smsl handling only */ + if (port_num == 0 && esp0 == FALSE) { + ib_port_info_set_master_smsl(p_pi, smsl); + goto Send; + } + + /* + PAST THIS POINT WE ARE HANDLING EITHER A NON PORT 0 OR ENHANCED PORT 0 + */ + if (ib_port_info_get_link_down_def_state(p_pi) != ib_port_info_get_link_down_def_state(p_old_pi)) send_set = TRUE; @@ -160,6 +220,20 @@ __osm_link_mgr_set_physp_pi(osm_sm_t * sm, sizeof(p_pi->master_sm_base_lid))) send_set = TRUE; + smsl = link_mgr_get_smsl(sm, p_physp); + if (smsl != ib_port_info_get_master_smsl(p_old_pi)) { + + ib_port_info_set_master_smsl(p_pi, smsl); + + OSM_LOG(sm->p_log, OSM_LOG_DEBUG, + "Setting SMSL to %d on GUID 0x%016" + PRIx64 ", port %d\n", + smsl, cl_ntoh64(osm_physp_get_port_guid(p_physp)), + port_num); + + send_set = TRUE; + } + p_pi->m_key_lease_period = sm->p_subn->opt.m_key_lease_period; if (memcmp(&p_pi->m_key_lease_period, @@ -290,6 +364,7 @@ __osm_link_mgr_set_physp_pi(osm_sm_t * sm, } } +Send: if (port_state != IB_LINK_NO_CHANGE && port_state != ib_port_info_get_port_state(p_old_pi)) { send_set = TRUE; diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index 5de496e..431977d 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -66,10 +66,8 @@ #include #include #include +#include -extern uint8_t osm_get_lash_sl(osm_opensm_t * p_osm, - const osm_port_t * p_src_port, - const osm_port_t * p_dst_port); typedef struct osm_pr_item { cl_list_item_t list_item; diff --git a/opensm/opensm/osm_ucast_lash.c b/opensm/opensm/osm_ucast_lash.c index 4e783bf..96a56e6 100644 --- a/opensm/opensm/osm_ucast_lash.c +++ b/opensm/opensm/osm_ucast_lash.c @@ -126,7 +126,7 @@ static void connect_switches(lash_t * p_lash, int sw1, int sw2, int phy_port_1) phy_port_1); } -static osm_switch_t *get_osm_switch_from_port(osm_port_t * port) +static osm_switch_t *get_osm_switch_from_port(const osm_port_t * port) { osm_physp_t *p = port->p_physp; if (p->p_node->sw) @@ -1274,8 +1274,8 @@ static void lash_delete(void *context) free(p_lash); } -uint8_t osm_get_lash_sl(osm_opensm_t * p_osm, osm_port_t * p_src_port, - osm_port_t * p_dst_port) +uint8_t osm_get_lash_sl(osm_opensm_t * p_osm, const osm_port_t * p_src_port, + const osm_port_t * p_dst_port) { unsigned dst_id; unsigned src_id; From sashak at voltaire.com Tue Mar 10 02:43:53 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Mar 2009 11:43:53 +0200 Subject: [ofa-general] Re: [PATCH][TRIVIAL] ibsim: Some cosmetic changes In-Reply-To: <20090306205055.GA6261@comcast.net> References: <20090306205055.GA6261@comcast.net> Message-ID: <20090310094346.GE23898@sashak.voltaire.com> On 15:50 Fri 06 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied with modification noted below. Thanks. > diff --git a/ibsim/sim_cmd.c b/ibsim/sim_cmd.c > index 94e0a14..bbd51b9 100644 > --- a/ibsim/sim_cmd.c > +++ b/ibsim/sim_cmd.c > @@ -274,10 +274,9 @@ static void port_change_lid(Port * port, int lid, int lmc) > if (lmc > 0) > port->lmc = lmc; > > - if (port->node->type == SWITCH_NODE) { > - if (port->node->sw) > - port->node->sw->portchange = 1; > - } else if (port->remotenode && port->remotenode->sw) > + if (port->node->type == SWITCH_NODE && port->node->sw) > + port->node->sw->portchange = 1; > + else if (port->remotenode && port->remotenode->sw) > port->remotenode->sw->portchange = 1; > } This is not equivalent. So I'm removing this chunk. Sasha From vlad at lists.openfabrics.org Tue Mar 10 03:21:50 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Tue, 10 Mar 2009 03:21:50 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090310-0200 daily build status Message-ID: <20090310102151.4310FE603C4@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From sashak at voltaire.com Tue Mar 10 04:40:50 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Mar 2009 13:40:50 +0200 Subject: [ofa-general] Re: [PATCH] opensm/osm_req.c: Shouldn't reveal port's MKey on Trap method In-Reply-To: <20090309110020.GA18067@comcast.net> References: <20090309110020.GA18067@comcast.net> Message-ID: <20090310114039.GA19978@sashak.voltaire.com> On 06:00 Mon 09 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Adding commit message which refers IBA 1.2.1 o14-1. Please try to provide informative commit message. Thanks. Sasha From sashak at voltaire.com Tue Mar 10 04:41:52 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Mar 2009 13:41:52 +0200 Subject: [ofa-general] Re: [PATCH] opensm/osm_inform.c: In __osm_send_report, make sure p_report_madw valid before using In-Reply-To: <20090309110117.GB18067@comcast.net> References: <20090309110117.GB18067@comcast.net> Message-ID: <20090310114152.GB19978@sashak.voltaire.com> On 06:01 Mon 09 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From nicolas.morey-chaisemartin at ext.bull.net Tue Mar 10 06:59:30 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Tue, 10 Mar 2009 14:59:30 +0100 Subject: [ofa-general] [PATCH 0/4] Enhanced Fat Tree Message-ID: <49B67242.5060605@ext.bull.net> Repost of the set of patches for minhop enhancement within the fat-tree algorithm. These patches are to be tested and discussed. They are not yet ready to be applied ! Nicolas Morey-Chaisemartin (4): Enhanced Fat-Tree algorithm: Added balanced minhop within Fat-Tree Added qsort for choosing secondary routes order instead of simple index Enhanced qsort compareason function so it looks at the next switch level. Replaced qsort by enhanced bubble sort Many fixes on balanced minhop enhancements Fixed stupid order on 2nd lvl sort Fixed bad init value for downgoing calls Fixed bug in port counters Fixed comments of bubble sort functions opensm/opensm/osm_ucast_ftree.c | 306 +++++++++++++++++++++++++++++++-------- 1 files changed, 244 insertions(+), 62 deletions(-) From nicolas.morey-chaisemartin at ext.bull.net Tue Mar 10 06:59:38 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Tue, 10 Mar 2009 14:59:38 +0100 Subject: [ofa-general] [PATCH 1/4] Enhanced Fat-Tree algorithm: Added balanced minhop within Fat-Tree Added qsort for choosing secondary routes order instead of simple index In-Reply-To: References: Message-ID: <49B6724A.1070800@ext.bull.net> Signed-off-by: Nicolas Morey-Chaisemartin --- opensm/opensm/osm_ucast_ftree.c | 169 ++++++++++++++++++++++++--------------- 1 files changed, 106 insertions(+), 63 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index 6e2e7de..7ca7f35 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -152,6 +152,7 @@ typedef struct ftree_port_group_t_ { boolean_t is_cn; /* whether this port is a compute node */ boolean_t is_io; /* whether this port is an I/O node */ uint32_t counter_down; /* number of allocated routs downwards */ + uint32_t counter_up; /* number of allocated routs downwards */ } ftree_port_group_t; /*************************************************** @@ -1899,6 +1900,27 @@ static void __osm_ftree_set_sw_fwd_table(IN cl_map_item_t * const p_map_item, p_sw->p_osm_sw); } + + +static int +__osm_ftree_port_group_compare_load_down(const void* p1,const void* p2){ + int temp = (*((ftree_port_group_t**)p1))->counter_down -(*((ftree_port_group_t**)p2))->counter_down ; + if(temp > 0) + return 1; + if(temp < 0) + return -1; + return 0; +} + +static int +__osm_ftree_port_group_compare_load_up(const void* p1,const void* p2){ + int temp = (*((ftree_port_group_t**)p1))->counter_up -(*((ftree_port_group_t**)p2))->counter_up ; + if(temp > 0) + return 1; + if(temp < 0) + return -1; + return 0; +} /*************************************************** ***************************************************/ @@ -1931,10 +1953,10 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, ftree_port_group_t *p_group; ftree_port_t *p_port; ftree_port_t *p_min_port; - uint16_t i; uint16_t j; uint16_t k; boolean_t created_route = FALSE; + boolean_t routed=0; /* we shouldn't enter here if both real_lid and main_path are false */ CL_ASSERT(is_real_lid || is_main_path); @@ -1944,11 +1966,13 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, return FALSE; /* foreach down-going port group (in indexing order) */ - i = p_sw->down_port_groups_idx; + + qsort(p_sw->down_port_groups,p_sw->down_port_groups_num,sizeof(*(p_sw->down_port_groups)), __osm_ftree_port_group_compare_load_up); + + for (k = 0; k < p_sw->down_port_groups_num; k++) { - p_group = p_sw->down_port_groups[i]; - i = (i + 1) % p_sw->down_port_groups_num; + p_group = p_sw->down_port_groups[k]; /* If this port group doesn't point to a switch, mark that the route was created and skip to the next group */ @@ -1998,6 +2022,10 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, cl_ntoh16(p_group->base_lid), __osm_ftree_tuple_to_str(p_sw->tuple), cl_ntoh16(p_group->remote_base_lid)); + /* We skip only if we have come through a longer path */ + if(((target_rank - highest_rank_in_route) + + (p_remote_sw->rank - highest_rank_in_route) + 2*reverse_hops) >= + osm_switch_get_least_hops(p_remote_sw->p_osm_sw, cl_ntoh16(target_lid))) continue; } @@ -2030,10 +2058,6 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, /* second case: skip the port group if the remote (lower) switch has been already configured for this target LID */ - if (is_real_lid && !is_main_path && - p_remote_sw->p_osm_sw->new_lft[cl_ntoh16(target_lid)] != - OSM_NO_PATH) - continue; /* setting fwd tbl port only if this is real LID */ if (is_real_lid) { @@ -2069,19 +2093,26 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, the upper side of the link (on switch with lower rank). Counter is promoted only if we're routing LID on the main path (whether it's a real LID or a dummy one). */ - if (is_main_path) - p_min_port->counter_up++; + /* if (is_main_path) + p_min_port->counter_up++;*/ /* Recursion step: Assign upgoing ports by stepping down, starting on REMOTE switch */ - created_route |= __osm_ftree_fabric_route_upgoing_by_going_down(p_ftree, p_remote_sw, /* remote switch - used as a route-upgoing alg. start point */ - NULL, /* prev. position - NULL to mark that we went down and not up */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - is_real_lid, /* whether the target LID is real or dummy */ - is_main_path, /* whether this is path to HCA that should by tracked by counters */ - highest_rank_in_route, reverse_hops); /* highest visited point in the tree before going down */ - } + routed = + __osm_ftree_fabric_route_upgoing_by_going_down(p_ftree, + p_remote_sw, /* remote switch - used as a route-upgoing alg. start point */ + NULL, /* prev. position - NULL to mark that we went down and not up */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + is_real_lid, /* whether the target LID is real or dummy */ + is_main_path, /* whether this is path to HCA that should by tracked by counters */ + highest_rank_in_route, reverse_hops); /* highest visited point in the tree before going down */ + created_route|=routed; + if(routed) + p_min_port->counter_up++; + p_group->counter_up++; + } + /* done scanning all the down-going port groups */ /* if the route was created, promote the index that @@ -2095,6 +2126,8 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, return created_route; } /* __osm_ftree_fabric_route_upgoing_by_going_down() */ + + /***************************************************/ /* @@ -2108,7 +2141,7 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, * assign-down-going-port-by-ascending-up on REMOTE switch (recursion) */ -static void +static boolean_t __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, IN ftree_sw_t * p_sw, IN ftree_sw_t * p_prev_sw, @@ -2127,12 +2160,14 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, ftree_port_t *p_min_port; uint16_t i; uint16_t j; + boolean_t created_route = FALSE; + boolean_t routed = FALSE; /* we shouldn't enter here if both real_lid and main_path are false */ CL_ASSERT(is_real_lid || is_main_path); /* Assign upgoing ports by stepping down, starting on THIS switch */ - __osm_ftree_fabric_route_upgoing_by_going_down(p_ftree, p_sw, /* local switch - used as a route-upgoing alg. start point */ + created_route = __osm_ftree_fabric_route_upgoing_by_going_down(p_ftree, p_sw, /* local switch - used as a route-upgoing alg. start point */ p_prev_sw, /* switch that we went up from (NULL means that we went down) */ target_lid, /* LID that we're routing to */ target_rank, /* rank of the LID that we're routing to */ @@ -2147,8 +2182,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* We go up by going down as we have some reverse_hop_credit left */ /* We use the index to scatter a bit the reverse up routes */ p_sw->down_port_groups_idx = - (p_sw->down_port_groups_idx + - 1) % p_sw->down_port_groups_num; + (p_sw->down_port_groups_idx + 1) % p_sw->down_port_groups_num; i = p_sw->down_port_groups_idx; for (j = 0; j < p_sw->down_port_groups_num; j++) { @@ -2156,39 +2190,30 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, i = (i + 1) % p_sw->down_port_groups_num; /* Skip this port group unless it points to a switch */ - if (p_group->remote_node_type != - IB_NODE_TYPE_SWITCH) + if (p_group->remote_node_type != IB_NODE_TYPE_SWITCH) continue; p_remote_sw = p_group->remote_hca_or_sw.p_sw; - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ - p_sw, /* this switch - prev. position switch for the function */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - is_real_lid, /* whether this target LID is real or dummy */ - is_main_path, /* whether this is path to HCA that should by tracked by counters */ - reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ - reverse_hops + 1); /* Number of reverse_hops done up to this point */ + created_route |=__osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + p_sw, /* this switch - prev. position switch for the function */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + is_real_lid, /* whether this target LID is real or dummy */ + is_main_path, /* whether this is path to HCA that should by tracked by counters */ + reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ + reverse_hops + 1); /* Number of reverse_hops done up to this point */ } } - return; + return created_route; } - /* Find the least loaded upgoing port group */ - p_min_group = NULL; - for (i = 0; i < p_sw->up_port_groups_num; i++) { - p_group = p_sw->up_port_groups[i]; - if (!p_min_group) { - /* first group that we're checking - use - it as a group with the lowest load */ - p_min_group = p_group; - } else if (p_group->counter_down < p_min_group->counter_down) { - /* this group is less loaded - use it as min */ - p_min_group = p_group; - } - } + /* We should generate a list of port sorted by load so we can find easily the least + * going port and explore the other pots on secondary routes more easily (and quickly) */ + qsort(p_sw->up_port_groups,p_sw->up_port_groups_num,sizeof(*(p_sw->up_port_groups)), __osm_ftree_port_group_compare_load_down); + + p_min_group = p_sw->up_port_groups[0]; /* Find the least loaded upgoing port in the selected group */ p_min_port = NULL; ports_num = (uint16_t) cl_ptr_vector_get_size(&p_min_group->ports); @@ -2293,7 +2318,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* Recursion step: Assign downgoing ports by stepping up, starting on REMOTE switch. */ - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + created_route|=__osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ p_sw, /* this switch - prev. position switch for the function */ target_lid, /* LID that we're routing to */ target_rank, /* rank of the LID that we're routing to */ @@ -2305,7 +2330,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* we're done for the third case */ if (!is_real_lid) - return; + return created_route; /* What's left to do at this point: * @@ -2339,14 +2364,15 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, * - go UP(TRUE,FALSE) to the remote switch */ - for (i = 0; i < p_sw->up_port_groups_num; i++) { + for (i = 1; i < p_sw->up_port_groups_num; i++) { p_group = p_sw->up_port_groups[i]; p_remote_sw = p_group->remote_hca_or_sw.p_sw; - /* skip if target lid has been already set on remote switch fwd tbl */ - if (p_remote_sw->p_osm_sw->new_lft[cl_ntoh16(target_lid)] != - OSM_NO_PATH) - continue; + /* skip if target lid has been already set on remote switch fwd tbl (with a bigger hop count)*/ + if (p_remote_sw->p_osm_sw->new_lft[cl_ntoh16(target_lid)] != OSM_NO_PATH) + if((target_rank -p_remote_sw->rank + 2*reverse_hops) >= + osm_switch_get_least_hops(p_remote_sw->p_osm_sw, cl_ntoh16(target_lid))) + continue; if (p_sw->is_leaf) { OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, @@ -2361,8 +2387,23 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, We can safely assume that switch will initiate very few traffic, so there's no point waisting runtime on trying to balance these routes - always pick port 0. */ + /* GET MIN PORT HERE */ + p_min_port = NULL; + ports_num = (uint16_t) cl_ptr_vector_get_size(&p_group->ports); + for (j = 0; j < ports_num; j++) { + cl_ptr_vector_at(&p_group->ports, j, (void *)&p_port); + if (!p_min_port) { + /* first port that we're checking - use + it as a port with the lowest load */ + p_min_port = p_port; + } else if (p_port->counter_down < p_min_port->counter_down) { + /* this port is less loaded - use it as min */ + p_min_port = p_port; + } + } - cl_ptr_vector_at(&p_group->ports, 0, (void *)&p_port); + p_port = p_min_port; + //cl_ptr_vector_at(&p_group->ports, 0, (void *)&p_port); p_remote_sw->p_osm_sw->new_lft[cl_ntoh16(target_lid)] = p_port->remote_port_num; @@ -2383,7 +2424,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* Recursion step: Assign downgoing ports by stepping up, starting on REMOTE switch. */ - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + routed = __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ p_sw, /* this switch - prev. position switch for the function */ target_lid, /* LID that we're routing to */ target_rank, /* rank of the LID that we're routing to */ @@ -2391,16 +2432,17 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, FALSE, /* whether this is path to HCA that should by tracked by counters */ reverse_hop_credit, /* Remaining reverse_hops allowed */ reverse_hops); /* Number of reverse_hops done up to this point */ + created_route|=routed; } - /* If we don't have any reverse hop credits, we are done */ - if (reverse_hop_credit == 0) - return; + /* If we don't have any reverse hop credits, we are done */ + if(reverse_hop_credit==0) + return created_route; - /* We explore all the down group ports */ - /* We try to reverse jump for each of them */ - /* They already have a route to us from the upgoing_by_going_down started earlier */ - /* This is only so it'll continue exploring up, after this step backwards */ + /* We explore all the down group ports */ + /* We try to reverse jump for each of them */ + /* They already have a route to us from the upgoing_by_going_down started earlier */ + /* This is only so it'll continue exploring up, after this step backwards*/ for (i = 0; i < p_sw->down_port_groups_num; i++) { p_group = p_sw->down_port_groups[i]; p_remote_sw = p_group->remote_hca_or_sw.p_sw; @@ -2411,7 +2453,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* Recursion step: Assign downgoing ports by stepping up, fter doing one step down starting on REMOTE switch. */ - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + created_route |=__osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ p_sw, /* this switch - prev. position switch for the function */ target_lid, /* LID that we're routing to */ target_rank, /* rank of the LID that we're routing to */ @@ -2420,6 +2462,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ reverse_hops + 1); /* Number of reverse_hops done up to this point */ } + return created_route; } /* ftree_fabric_route_downgoing_by_going_up() */ -- 1.6.2-rc2.GIT From nicolas.morey-chaisemartin at ext.bull.net Tue Mar 10 06:59:45 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Tue, 10 Mar 2009 14:59:45 +0100 Subject: [ofa-general] [PATCH 2/4] Enhanced qsort compareason function so it looks at the next switch level. In-Reply-To: References: Message-ID: <49B67251.4080803@ext.bull.net> Signed-off-by: Nicolas Morey-Chaisemartin --- opensm/opensm/osm_ucast_ftree.c | 56 ++++++++++++++++++++++++++++++++++++--- 1 files changed, 52 insertions(+), 4 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index 7ca7f35..0ba4999 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -173,6 +173,7 @@ typedef struct ftree_sw_t_ { uint8_t up_port_groups_num; boolean_t is_leaf; unsigned down_port_groups_idx; + uint32_t min_counter_down; } ftree_sw_t; /*************************************************** @@ -1900,8 +1901,33 @@ static void __osm_ftree_set_sw_fwd_table(IN cl_map_item_t * const p_map_item, p_sw->p_osm_sw); } +static void +__osm_ftree_recalculate_min_counter_down(ftree_sw_t *p_sw){ + uint32_t min= (1<<30); + uint32_t i; + for(i=0;i < p_sw->down_port_groups_num; i++) { + if(p_sw->down_port_groups[i]->counter_down < min){ + min = p_sw->down_port_groups[i]->counter_down; + } + } + p_sw->min_counter_down = min; + return; +} +static uint32_t +__osm_ftree_find_lowest_loaded_group_on_sw(ftree_sw_t *p_sw){ + /*uint32_t min= (1<<30); + uint32_t i; + for(i=0;i < p_sw->down_port_groups_num; i++) { + if(p_sw->down_port_groups[i]->counter_down < min){ + min = p_sw->down_port_groups[i]->counter_down; + } + }*/ + return p_sw->min_counter_down; +} +/* This is for downgoing_by_going_up. + * If load are equals, let's have a look at the remote switches and find the less loaded one */ static int __osm_ftree_port_group_compare_load_down(const void* p1,const void* p2){ int temp = (*((ftree_port_group_t**)p1))->counter_down -(*((ftree_port_group_t**)p2))->counter_down ; @@ -1909,9 +1935,26 @@ __osm_ftree_port_group_compare_load_down(const void* p1,const void* p2){ return 1; if(temp < 0) return -1; - return 0; + + /* Find the less loaded remote sw and choose this one */ + do{ + uint32_t load1=__osm_ftree_find_lowest_loaded_group_on_sw((*((ftree_port_group_t**)p1))->remote_hca_or_sw.p_sw); + uint32_t load2=__osm_ftree_find_lowest_loaded_group_on_sw((*((ftree_port_group_t**)p2))->remote_hca_or_sw.p_sw); + temp = load1-load2; + if(temp > 0) + return 1; + if(temp < 0) + return -1; + }while(0); + + /* If they are both equal, choose the biggest GUID */ + if(((*((ftree_port_group_t**)p1)))->remote_port_guid > ((*((ftree_port_group_t**)p2)))->remote_port_guid) + return 1; + + return -1; } +/* This is for upgoing_by_going_down. There is not much equilibration to do so don't bother looking at the next rank */ static int __osm_ftree_port_group_compare_load_up(const void* p1,const void* p2){ int temp = (*((ftree_port_group_t**)p1))->counter_up -(*((ftree_port_group_t**)p2))->counter_up ; @@ -2212,8 +2255,8 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, * going port and explore the other pots on secondary routes more easily (and quickly) */ qsort(p_sw->up_port_groups,p_sw->up_port_groups_num,sizeof(*(p_sw->up_port_groups)), __osm_ftree_port_group_compare_load_down); - p_min_group = p_sw->up_port_groups[0]; + /* Find the least loaded upgoing port in the selected group */ p_min_port = NULL; ports_num = (uint16_t) cl_ptr_vector_get_size(&p_min_group->ports); @@ -2277,12 +2320,17 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, __osm_ftree_tuple_to_str(p_sw->tuple), __osm_ftree_tuple_to_str(p_remote_sw->tuple)); } + /* The number of downgoing routes is tracked in the p_group->counter_down p_port->counter_down counters of the group and port that belong to the lower side of the link (on switch with higher rank) */ p_min_group->counter_down++; p_min_port->counter_down++; + if(p_min_group->counter_down == (p_min_group->remote_hca_or_sw.p_sw->min_counter_down+1)){ + __osm_ftree_recalculate_min_counter_down(p_min_group->remote_hca_or_sw.p_sw); + } + if (is_real_lid) { /* This LID may already be in the LFT in the reverse_hop feature is used */ /* We update the LFT only if this LID isn't already present. */ @@ -2371,7 +2419,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* skip if target lid has been already set on remote switch fwd tbl (with a bigger hop count)*/ if (p_remote_sw->p_osm_sw->new_lft[cl_ntoh16(target_lid)] != OSM_NO_PATH) if((target_rank -p_remote_sw->rank + 2*reverse_hops) >= - osm_switch_get_least_hops(p_remote_sw->p_osm_sw, cl_ntoh16(target_lid))) + osm_switch_get_least_hops(p_remote_sw->p_osm_sw, cl_ntoh16(target_lid))) continue; if (p_sw->is_leaf) { @@ -2387,7 +2435,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, We can safely assume that switch will initiate very few traffic, so there's no point waisting runtime on trying to balance these routes - always pick port 0. */ - /* GET MIN PORT HERE */ + p_min_port = NULL; ports_num = (uint16_t) cl_ptr_vector_get_size(&p_group->ports); for (j = 0; j < ports_num; j++) { -- 1.6.2-rc2.GIT From nicolas.morey-chaisemartin at ext.bull.net Tue Mar 10 06:59:53 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Tue, 10 Mar 2009 14:59:53 +0100 Subject: [ofa-general] [PATCH 3/4] Replaced qsort by enhanced bubble sort In-Reply-To: References: Message-ID: <49B67259.3080804@ext.bull.net> Signed-off-by: Nicolas Morey-Chaisemartin --- opensm/opensm/osm_ucast_ftree.c | 107 ++++++++++++++++++++++++++------------- 1 files changed, 72 insertions(+), 35 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index 0ba4999..bfefa0f 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -81,7 +81,6 @@ typedef enum { ** Forward references ** ***************************************************/ - struct ftree_sw_t_; struct ftree_hca_t_; struct ftree_port_t_; @@ -174,6 +173,7 @@ typedef struct ftree_sw_t_ { boolean_t is_leaf; unsigned down_port_groups_idx; uint32_t min_counter_down; + boolean_t counter_up_changed; } ftree_sw_t; /*************************************************** @@ -1901,7 +1901,7 @@ static void __osm_ftree_set_sw_fwd_table(IN cl_map_item_t * const p_map_item, p_sw->p_osm_sw); } -static void +static inline void __osm_ftree_recalculate_min_counter_down(ftree_sw_t *p_sw){ uint32_t min= (1<<30); uint32_t i; @@ -1914,23 +1914,16 @@ __osm_ftree_recalculate_min_counter_down(ftree_sw_t *p_sw){ return; } -static uint32_t +static inline uint32_t __osm_ftree_find_lowest_loaded_group_on_sw(ftree_sw_t *p_sw){ - /*uint32_t min= (1<<30); - uint32_t i; - for(i=0;i < p_sw->down_port_groups_num; i++) { - if(p_sw->down_port_groups[i]->counter_down < min){ - min = p_sw->down_port_groups[i]->counter_down; - } - }*/ return p_sw->min_counter_down; } /* This is for downgoing_by_going_up. * If load are equals, let's have a look at the remote switches and find the less loaded one */ -static int -__osm_ftree_port_group_compare_load_down(const void* p1,const void* p2){ - int temp = (*((ftree_port_group_t**)p1))->counter_down -(*((ftree_port_group_t**)p2))->counter_down ; +static inline int +__osm_ftree_port_group_compare_load_down(const ftree_port_group_t *p1,const ftree_port_group_t *p2){ + int temp = p1->counter_down -p2->counter_down ; if(temp > 0) return 1; if(temp < 0) @@ -1938,32 +1931,78 @@ __osm_ftree_port_group_compare_load_down(const void* p1,const void* p2){ /* Find the less loaded remote sw and choose this one */ do{ - uint32_t load1=__osm_ftree_find_lowest_loaded_group_on_sw((*((ftree_port_group_t**)p1))->remote_hca_or_sw.p_sw); - uint32_t load2=__osm_ftree_find_lowest_loaded_group_on_sw((*((ftree_port_group_t**)p2))->remote_hca_or_sw.p_sw); + uint32_t load1=__osm_ftree_find_lowest_loaded_group_on_sw(p1->remote_hca_or_sw.p_sw); + uint32_t load2=__osm_ftree_find_lowest_loaded_group_on_sw(p2->remote_hca_or_sw.p_sw); temp = load1-load2; if(temp > 0) return 1; - if(temp < 0) - return -1; + return 0; }while(0); - /* If they are both equal, choose the biggest GUID */ - if(((*((ftree_port_group_t**)p1)))->remote_port_guid > ((*((ftree_port_group_t**)p2)))->remote_port_guid) - return 1; +} - return -1; +static inline void +__osm_ftree_bubble_sort_up(ftree_port_group_t **p_group_array, uint32_t nmemb) +{ + uint32_t i = 0; /* Indice de répétition du tri */ + uint32_t j = 0; /* Variable de boucle */ + ftree_port_group_t *tmp = p_group_array[0]; /* Variable de stockage temporaire et de fin de tri*/ + if(tmp->hca_or_sw.p_sw->counter_up_changed == FALSE){ + return; + } + /* Boucle de répétition du tri et le test qui + arrête le tri dès que le tableau est ordonné */ + for(i = 0 ; tmp ; i++) + { + /* Supposons le tableau ordonné */ + tmp = NULL; + /* Vérification des éléments des places j et j-1 */ + for(j = 1 ; j < (nmemb - i) ; j++) + { + /* Si les 2 éléments sont mal triés */ + if( p_group_array[j]->counter_up < p_group_array[j-1]->counter_up) + { + /* Inversion des 2 éléments */ + tmp = p_group_array[j-1]; + p_group_array[j-1] = p_group_array[j]; + p_group_array[j] = tmp; + + } + } + } + p_group_array[0]->hca_or_sw.p_sw->counter_up_changed = FALSE; } -/* This is for upgoing_by_going_down. There is not much equilibration to do so don't bother looking at the next rank */ -static int -__osm_ftree_port_group_compare_load_up(const void* p1,const void* p2){ - int temp = (*((ftree_port_group_t**)p1))->counter_up -(*((ftree_port_group_t**)p2))->counter_up ; - if(temp > 0) - return 1; - if(temp < 0) - return -1; - return 0; +static inline void +__osm_ftree_bubble_sort_down(ftree_port_group_t **p_group_array, uint32_t nmemb) +{ + uint32_t i = 0; /* Indice de répétition du tri */ + uint32_t j = 0; /* Variable de boucle */ + ftree_port_group_t *tmp = p_group_array[0]; ; /* Variable de stockage temporaire et de fin de tri*/ + + /* Boucle de répétition du tri et le test qui + arrête le tri dès que le tableau est ordonné */ + for(i = 0 ; tmp ; i++) + { + /* Supposons le tableau ordonné */ + tmp = NULL; + /* Vérification des éléments des places j et j-1 */ + for(j = 1 ; j < (nmemb - i) ; j++) + { + /* Si les 2 éléments sont mal triés */ + if( __osm_ftree_port_group_compare_load_down(p_group_array[j],p_group_array[j-1]) < 0 ) + { + /* Inversion des 2 éléments */ + tmp = p_group_array[j-1]; + p_group_array[j-1] = p_group_array[j]; + p_group_array[j] = tmp; + + } + } + } } + + /*************************************************** ***************************************************/ @@ -2009,10 +2048,7 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, return FALSE; /* foreach down-going port group (in indexing order) */ - - qsort(p_sw->down_port_groups,p_sw->down_port_groups_num,sizeof(*(p_sw->down_port_groups)), __osm_ftree_port_group_compare_load_up); - - + __osm_ftree_bubble_sort_up(p_sw->down_port_groups,p_sw->down_port_groups_num); for (k = 0; k < p_sw->down_port_groups_num; k++) { p_group = p_sw->down_port_groups[k]; @@ -2154,6 +2190,7 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, if(routed) p_min_port->counter_up++; p_group->counter_up++; + p_group->hca_or_sw.p_sw->counter_up_changed = TRUE; } /* done scanning all the down-going port groups */ @@ -2253,7 +2290,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* We should generate a list of port sorted by load so we can find easily the least * going port and explore the other pots on secondary routes more easily (and quickly) */ - qsort(p_sw->up_port_groups,p_sw->up_port_groups_num,sizeof(*(p_sw->up_port_groups)), __osm_ftree_port_group_compare_load_down); + __osm_ftree_bubble_sort_down(p_sw->up_port_groups,p_sw->up_port_groups_num); p_min_group = p_sw->up_port_groups[0]; -- 1.6.2-rc2.GIT From nicolas.morey-chaisemartin at ext.bull.net Tue Mar 10 07:00:14 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Tue, 10 Mar 2009 15:00:14 +0100 Subject: [ofa-general] [PATCH 4/4] Many fixes on balanced minhop enhancements In-Reply-To: References: Message-ID: <49B6726E.5080706@ext.bull.net> Fixed stupid order on 2nd lvl sort Fixed bad init value for downgoing calls Fixed bug in port counters Fixed comments of bubble sort functions Signed-off-by: Nicolas Morey-Chaisemartin --- opensm/opensm/osm_ucast_ftree.c | 106 +++++++++++++++++++++++++++++---------- 1 files changed, 80 insertions(+), 26 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index bfefa0f..638504c 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -1901,6 +1901,13 @@ static void __osm_ftree_set_sw_fwd_table(IN cl_map_item_t * const p_map_item, p_sw->p_osm_sw); } +/*************************************************** + ***************************************************/ + +/* + * Function: Finds the least loaded port group and stores its counter + * Given : A switch + */ static inline void __osm_ftree_recalculate_min_counter_down(ftree_sw_t *p_sw){ uint32_t min= (1<<30); @@ -1914,13 +1921,23 @@ __osm_ftree_recalculate_min_counter_down(ftree_sw_t *p_sw){ return; } +/* + * Function: Return the counter value of the least loaded down port group + * Given : A switch + */ static inline uint32_t __osm_ftree_find_lowest_loaded_group_on_sw(ftree_sw_t *p_sw){ return p_sw->min_counter_down; } -/* This is for downgoing_by_going_up. - * If load are equals, let's have a look at the remote switches and find the less loaded one */ +/* + * Function: Compare the load of two port groups and return which is the least loaded + * Given : Two port groups with remote switch + * When both port groups are equally loaded, it picks the one whom + * remote switch down ports are least loaded. + * This way, it prefers the switch from where it will be easier to go down (creating upward routes). + * If both are equal, it picks the bigger GUID to be deterministic. + */ static inline int __osm_ftree_port_group_compare_load_down(const ftree_port_group_t *p1,const ftree_port_group_t *p2){ int temp = p1->counter_down -p2->counter_down ; @@ -1936,63 +1953,100 @@ __osm_ftree_port_group_compare_load_down(const ftree_port_group_t *p1,const ftre temp = load1-load2; if(temp > 0) return 1; - return 0; }while(0); + /* If they are both equal, choose the biggest GUID */ + if(p1->remote_port_guid > p2->remote_port_guid) + return 1; + + return -1; } +/* + * Function: Sorts an array of port group by up load order + * Given : A port group array and its length + * As the list is mostly sorted, we used a bubble sort instead of qsort + * as it is much faster. + * + * Important note: + * This function and __osm_ftree_bubble_sort_down must NOT be factorized. + * Although most of the code is the same and a function pointer could be used + * for the compareason function, it would prevent the compareason function to be inlined + * and cost a great deal to performances. + */ static inline void __osm_ftree_bubble_sort_up(ftree_port_group_t **p_group_array, uint32_t nmemb) { - uint32_t i = 0; /* Indice de répétition du tri */ - uint32_t j = 0; /* Variable de boucle */ - ftree_port_group_t *tmp = p_group_array[0]; /* Variable de stockage temporaire et de fin de tri*/ + uint32_t i = 0; + uint32_t j = 0; + ftree_port_group_t *tmp = p_group_array[0]; + + /* As this function is a great number of times, we only go into the loop + * if one of the port counters has changed, thus saving some tests */ if(tmp->hca_or_sw.p_sw->counter_up_changed == FALSE){ return; } - /* Boucle de répétition du tri et le test qui - arrête le tri dès que le tableau est ordonné */ + /* While we did modifications on the array order*/ + /* i may grew above array length but next loop will fail and tmp will be null for the next time + * this way we save a test i < nmemb for each pass through the loop */ for(i = 0 ; tmp ; i++) { - /* Supposons le tableau ordonné */ + /* Assume the array is orderd */ tmp = NULL; - /* Vérification des éléments des places j et j-1 */ + /* Comparing elements j and j-1 */ for(j = 1 ; j < (nmemb - i) ; j++) { - /* Si les 2 éléments sont mal triés */ + /* If they are the wrong way around */ if( p_group_array[j]->counter_up < p_group_array[j-1]->counter_up) { - /* Inversion des 2 éléments */ + /* We invert them */ tmp = p_group_array[j-1]; p_group_array[j-1] = p_group_array[j]; p_group_array[j] = tmp; - + /* This sets tmp != NULL so the main loop will make another pass */ } } } + + /* We have reordered the array so as long noone changes the counter + * it's not necessary to do it again */ p_group_array[0]->hca_or_sw.p_sw->counter_up_changed = FALSE; } +/* + * Function: Sorts an array of port group. Order is decide through + * __osm_ftree_port_group_compare_load_down ( up counters, least load remote switch, biggest GUID) + * Given : A port group array and its length. Each port group points to a remote switch (not a HCA) + * As the list is mostly sorted, we used a bubble sort instead of qsort + * as it is much faster. + * + * Important note: + * This function and __osm_ftree_bubble_sort_up must NOT be factorized. + * Although most of the code is the same and a function pointer could be used + * for the compareason function, it would prevent the compareason function to be inlined + * and cost a great deal to performances. + */ static inline void __osm_ftree_bubble_sort_down(ftree_port_group_t **p_group_array, uint32_t nmemb) { - uint32_t i = 0; /* Indice de répétition du tri */ - uint32_t j = 0; /* Variable de boucle */ - ftree_port_group_t *tmp = p_group_array[0]; ; /* Variable de stockage temporaire et de fin de tri*/ + uint32_t i = 0; + uint32_t j = 0; + ftree_port_group_t *tmp = p_group_array[0]; - /* Boucle de répétition du tri et le test qui - arrête le tri dès que le tableau est ordonné */ + /* While we did modifications on the array order*/ + /* i may grew above array length but next loop will fail and tmp will be null for the next time + * this way we save a test i < nmemb for each pass through the loop */ for(i = 0 ; tmp ; i++) { - /* Supposons le tableau ordonné */ - tmp = NULL; - /* Vérification des éléments des places j et j-1 */ + /* Assume the array is orderd */ + tmp = NULL; + /* Comparing elements j and j-1 */ for(j = 1 ; j < (nmemb - i) ; j++) { - /* Si les 2 éléments sont mal triés */ + /* If they are the wrong way around */ if( __osm_ftree_port_group_compare_load_down(p_group_array[j],p_group_array[j-1]) < 0 ) { - /* Inversion des 2 éléments */ + /* We invert them */ tmp = p_group_array[j-1]; p_group_array[j-1] = p_group_array[j]; p_group_array[j] = tmp; @@ -2187,12 +2241,12 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, is_main_path, /* whether this is path to HCA that should by tracked by counters */ highest_rank_in_route, reverse_hops); /* highest visited point in the tree before going down */ created_route|=routed; - if(routed) + if(routed){ p_min_port->counter_up++; p_group->counter_up++; p_group->hca_or_sw.p_sw->counter_up_changed = TRUE; } - + } /* done scanning all the down-going port groups */ /* if the route was created, promote the index that @@ -2449,7 +2503,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, * - go UP(TRUE,FALSE) to the remote switch */ - for (i = 1; i < p_sw->up_port_groups_num; i++) { + for (i = 0; i < p_sw->up_port_groups_num; i++) { p_group = p_sw->up_port_groups[i]; p_remote_sw = p_group->remote_hca_or_sw.p_sw; -- 1.6.2-rc2.GIT From swise at opengridcomputing.com Tue Mar 10 07:23:25 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Tue, 10 Mar 2009 09:23:25 -0500 Subject: [ofa-general] RE: tcp/rdma port unification patch in librdmacm In-Reply-To: <49B61C5D.6060900@Voltaire.com> References: <49B17E15.4050406@opengridcomputing.com> <811553D6D34A4E6F9AB686CA2E441D35@amr.corp.intel.com> <49B61C5D.6060900@Voltaire.com> Message-ID: <49B677DD.4020300@opengridcomputing.com> Or Gerlitz wrote: > Sean Hefty wrote: > >> I'm undecided on the best way to expose this. Always enabled? Enabled during >> the build? Enabled through a new API? Other? >> > > Hi Sean, > > Experimenting with the kernel patch (below) applied on Linus tree, such that the rdma_cm module is loaded with the param being ON and running rping, I didn't see any extra socket listed in the netstat -natp output. So to start with (I tend to think it was explained in the past... sorry for asking again) - is it correct that the approach taken by this patch is to consume only a --port-- from the tcp port space but not to have actually a visible socket? is this approach applicable to user space? > Linux netstat doesn't show TCP sockets that are bound but not yet listening nor connecting/connected. Steve. From sashak at voltaire.com Tue Mar 10 07:50:54 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Mar 2009 16:50:54 +0200 Subject: [ofa-general] Re: [PATCH] libibmad/rpc.c: Handle redirection status In-Reply-To: <20090309135401.GA21411@comcast.net> References: <20090309135401.GA21411@comcast.net> Message-ID: <20090310145054.GC19978@sashak.voltaire.com> Hi Hal, On 08:54 Mon 09 Mar , Hal Rosenstock wrote: > > Also, in mad_rpc, status should be based on management class Please try to not mix two things in one patch. > @@ -185,10 +186,28 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport > > mad = umad_get_mad(rcvbuf); > > - if ((status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F)) != 0) { > - ERRS("MAD completed with error status 0x%x; dport (%s)", > - status, portid2str(dport)); > - return 0; > + mgmtclass = mad_get_field(mad, 0, IB_MAD_MGMTCLASS_F); > + if (mgmtclass == 1 || mgmtclass == 0x81) { > + if (mgmtclass == 1) > + status = mad_get_field(mad, 0, IB_MAD_STATUS_F); > + else > + status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F); > + if (status != 0) { > + ERRS("MAD completed with error status 0x%x; dport (%s)", > + status, portid2str(dport)); > + return 0; > + } Basically it is about the same field. Why do we need such flow? > + } else { > + if ((status = mad_get_field(mad, 0, IB_MAD_STATUS_F)) != 0) { } else if (...) { > + if (status & 2) { /* redirection */ > + ERRS("MAD redirection not supported; dport (%s)", > + portid2str(dport)); > + } else { > + ERRS("MAD completed with error status 0x%x; dport (%s)", > + status, portid2str(dport)); > + } You are sending patches which cleans {} around single expressions... Just wrap ERRS() macro with 'do {..} while (0)' and code will be cleaner. > + return 0; > + } > } > > if (ibdebug) { > @@ -225,8 +244,13 @@ void *mad_rpc_rmpp(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * > mad = umad_get_mad(rcvbuf); > > if ((status = mad_get_field(mad, 0, IB_MAD_STATUS_F)) != 0) { > - ERRS("MAD completed with error status 0x%x; dport (%s)", > - status, portid2str(dport)); > + if (status & 2) { /* redirection */ > + ERRS("MAD redirection not supported; dport (%s)", > + portid2str(dport)); > + } else { > + ERRS("MAD completed with error status 0x%x; dport (%s)", > + status, portid2str(dport)); > + } Ditto. Sasha From hal.rosenstock at gmail.com Tue Mar 10 07:55:44 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 10 Mar 2009 09:55:44 -0500 Subject: [ofa-general] Re: [PATCH] libibmad/rpc.c: Handle redirection status In-Reply-To: <20090310145054.GC19978@sashak.voltaire.com> References: <20090309135401.GA21411@comcast.net> <20090310145054.GC19978@sashak.voltaire.com> Message-ID: Sasha, On Tue, Mar 10, 2009 at 9:50 AM, Sasha Khapyorsky wrote: > Hi Hal, > > On 08:54 Mon 09 Mar     , Hal Rosenstock wrote: >> >> Also, in mad_rpc, status should be based on management class > > Please try to not mix two things in one patch. I can break this into two if that is your preference. >> @@ -185,10 +186,28 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport >> >>       mad = umad_get_mad(rcvbuf); >> >> -     if ((status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F)) != 0) { >> -             ERRS("MAD completed with error status 0x%x; dport (%s)", >> -                  status, portid2str(dport)); >> -             return 0; >> +     mgmtclass = mad_get_field(mad, 0, IB_MAD_MGMTCLASS_F); >> +     if (mgmtclass == 1 || mgmtclass == 0x81) { >> +             if (mgmtclass == 1) >> +                     status = mad_get_field(mad, 0, IB_MAD_STATUS_F); >> +             else >> +                     status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F); >> +             if (status != 0) { >> +                     ERRS("MAD completed with error status 0x%x; dport (%s)", >> +                          status, portid2str(dport)); >> +                     return 0; >> +             } > > Basically it is about the same field. One bit difference. > Why do we need such flow? Strict spec compliance. >> +     } else { >> +             if ((status = mad_get_field(mad, 0, IB_MAD_STATUS_F)) != 0) { > >        } else if (...) { > >> +                     if (status & 2) { /* redirection */ >> +                             ERRS("MAD redirection not supported; dport (%s)", >> +                                  portid2str(dport)); >> +                     } else { >> +                             ERRS("MAD completed with error status 0x%x; dport (%s)", >> +                                  status, portid2str(dport)); >> +                     } > > You are sending patches which cleans {} around single expressions... > Just wrap ERRS() macro with 'do {..} while (0)' and code will be > cleaner. I'm not following what you mean by this. -- Hal >> +                     return 0; >> +             } >>       } >> >>       if (ibdebug) { >> @@ -225,8 +244,13 @@ void *mad_rpc_rmpp(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * >>       mad = umad_get_mad(rcvbuf); >> >>       if ((status = mad_get_field(mad, 0, IB_MAD_STATUS_F)) != 0) { >> -             ERRS("MAD completed with error status 0x%x; dport (%s)", >> -                  status, portid2str(dport)); >> +             if (status & 2) { /* redirection */ >> +                     ERRS("MAD redirection not supported; dport (%s)", >> +                          portid2str(dport)); >> +             } else { >> +                     ERRS("MAD completed with error status 0x%x; dport (%s)", >> +                          status, portid2str(dport)); >> +             } > > Ditto. > > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hal.rosenstock at gmail.com Tue Mar 10 07:56:43 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 10 Mar 2009 09:56:43 -0500 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH][TRIVIAL] ibsim: Some cosmetic changes In-Reply-To: <20090310094346.GE23898@sashak.voltaire.com> References: <20090306205055.GA6261@comcast.net> <20090310094346.GE23898@sashak.voltaire.com> Message-ID: On Tue, Mar 10, 2009 at 4:43 AM, Sasha Khapyorsky wrote: > On 15:50 Fri 06 Mar     , Hal Rosenstock wrote: >> >> Signed-off-by: Hal Rosenstock > > Applied with modification noted below. Thanks. > >> diff --git a/ibsim/sim_cmd.c b/ibsim/sim_cmd.c >> index 94e0a14..bbd51b9 100644 >> --- a/ibsim/sim_cmd.c >> +++ b/ibsim/sim_cmd.c >> @@ -274,10 +274,9 @@ static void port_change_lid(Port * port, int lid, int lmc) >>       if (lmc > 0) >>               port->lmc = lmc; >> >> -     if (port->node->type == SWITCH_NODE) { >> -             if (port->node->sw) >> -                     port->node->sw->portchange = 1; >> -     } else if (port->remotenode && port->remotenode->sw) >> +     if (port->node->type == SWITCH_NODE && port->node->sw) >> +             port->node->sw->portchange = 1; >> +     else if (port->remotenode && port->remotenode->sw) >>               port->remotenode->sw->portchange = 1; >>  } > > This is not equivalent. Why not ? -- Hal > So I'm removing this chunk. > > Sasha > - Show quoted text - > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hal.rosenstock at gmail.com Tue Mar 10 08:13:12 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 10 Mar 2009 10:13:12 -0500 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH v2 1/2] opensm: Handle trap repress on trap 144 generation In-Reply-To: <20090310093154.GD23898@sashak.voltaire.com> References: <20090306181948.GA1706@comcast.net> <20090310093154.GD23898@sashak.voltaire.com> Message-ID: On Tue, Mar 10, 2009 at 4:31 AM, Sasha Khapyorsky wrote: > - Show quoted text - > On 13:19 Fri 06 Mar     , Hal Rosenstock wrote: >> diff --git a/opensm/opensm/osm_trap_rcv.c b/opensm/opensm/osm_trap_rcv.c >> index 6639246..4576a8b 100644 >> --- a/opensm/opensm/osm_trap_rcv.c >> +++ b/opensm/opensm/osm_trap_rcv.c >> @@ -2,6 +2,7 @@ >>   * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. >>   * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. >>   * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. >> + * Copyright (c) 2009 HNR Consulting. All rights reserved. >>   * >>   * This software is available to you under a choice of one of two >>   * licenses.  You may choose to be licensed under the terms of the GNU >> @@ -649,18 +650,34 @@ Exit: >>  } >> >>  /********************************************************************** >> - CURRENTLY WE ARE NOT CREATING TRAPS - SO THIS CALL IN AN ERROR >>  **********************************************************************/ >>  static void >>  trap_rcv_process_response(IN osm_sm_t * sm, >>                         IN const osm_madw_t * const p_madw) >>  { >> +     uint8_t payload[sizeof(ib_mad_notice_attr_t)]; >> +     ib_smp_t *p_smp; >> +     ib_mad_notice_attr_t *p_ntci = (ib_mad_notice_attr_t *) payload; >> + >> +     CL_ASSERT(p_madw); >> + >> +     if (!osm_log_is_active(sm->p_log, OSM_LOG_VERBOSE)) >> +             return; >> >>       OSM_LOG_ENTER(sm->p_log); >> >> -     OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3808: " >> -             "This function is not supported yet\n"); >> +     if (p_madw->p_mad->mgmt_class != IB_MCLASS_SUBN_LID && >> +         p_madw->p_mad->mgmt_class != IB_MCLASS_SUBN_DIR) >> +             goto Exit; >> >> +     p_smp = osm_madw_get_smp_ptr(p_madw); >> + >> +     memset(payload, 0, sizeof(payload)); >> +     memcpy(payload, &p_smp->data, IB_SMP_DATA_SIZE); >> + >> +     osm_dump_notice(sm->p_log, p_ntci, OSM_LOG_VERBOSE); >> + >> +Exit: >>       OSM_LOG_EXIT(sm->p_log); >>  } > > Almost fine, but one thing still be unclear for me. How is this function > trap_rcv_process_response() reachable and what is this supposed to do? It isn't anymore; I'll remove it in next version. -- Hal > Sasha > - Show quoted text - > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From sashak at voltaire.com Tue Mar 10 08:18:32 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Mar 2009 17:18:32 +0200 Subject: [ofa-general] Re: [PATCH][TRIVIAL] ibsim: Some cosmetic changes In-Reply-To: References: <20090306205055.GA6261@comcast.net> <20090310094346.GE23898@sashak.voltaire.com> Message-ID: <20090310151832.GD19978@sashak.voltaire.com> On 09:56 Tue 10 Mar , Hal Rosenstock wrote: > > > > This is not equivalent. > > Why not ? In the original code: if (cond1) { if (cond2) do1(); } else do2(); do2() will work *only* when cond1 is false. After your modification: if (cond1 && cond2) do1(); else do2(); do2() will work when cond1 *or* cond2 are false. Sasha From tziporet at dev.mellanox.co.il Tue Mar 10 08:16:49 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Tue, 10 Mar 2009 17:16:49 +0200 Subject: [ofa-general] ***SPAM*** problem about the OpenSM In-Reply-To: <20090309113918.167a1b10.weiny2@llnl.gov> References: <91fe68d50903090441t114d70abv2f5091d00d3d185b@mail.gmail.com> <20090309113918.167a1b10.weiny2@llnl.gov> Message-ID: <49B68461.5070201@mellanox.co.il> Ira Weiny wrote: > > I don't know the answer to this as I don't know how OpenSM is installed, if at > all, on a Mellanox Switch. > There are switches of IS4 with CPU that allow you to run OSM on Mellanox switch. Do you have such a switch? Tziporet From sashak at voltaire.com Tue Mar 10 08:22:38 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Mar 2009 17:22:38 +0200 Subject: [ofa-general] Re: [PATCHv3] Add pkey table support to osm_get_all_port_attr In-Reply-To: <20090306123038.GA28792@comcast.net> References: <20090306123038.GA28792@comcast.net> Message-ID: <20090310152230.GE19978@sashak.voltaire.com> On 07:30 Fri 06 Mar , Hal Rosenstock wrote: > > Only supported in osm_vendor_ibumad.c (separate patch for other > vendor layers) > Also, update applications using this (osmtest, opensm) > > Signed-off-by: Hal Rosenstock Applied. Thanks. Small note is below. > @@ -73,6 +74,8 @@ boolean_t osmt_is_debug(void) > > /********************************************************************** > **********************************************************************/ > +void show_usage(void); > + > void show_usage() > { > printf > @@ -207,13 +210,19 @@ void show_usage() > > /********************************************************************** > **********************************************************************/ > +static void print_all_guids(IN osmtest_t * p_osmt); > static void print_all_guids(IN osmtest_t * p_osmt) > { Those prototypes are not needed, I'm removing... Sasha From sashak at voltaire.com Tue Mar 10 08:27:17 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Mar 2009 17:27:17 +0200 Subject: [ofa-general] Re: [PATCH v2 1/2] opensm: Handle trap repress on trap 144 generation In-Reply-To: References: <20090306181948.GA1706@comcast.net> <20090310093154.GD23898@sashak.voltaire.com> Message-ID: <20090310152717.GF19978@sashak.voltaire.com> On 10:13 Tue 10 Mar , Hal Rosenstock wrote: > > > > Almost fine, but one thing still be unclear for me. How is this function > > trap_rcv_process_response() reachable and what is this supposed to do? > > It isn't anymore; I'll remove it in next version. No need. I will remove by myself. Sasha From sashak at voltaire.com Tue Mar 10 08:37:54 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Mar 2009 17:37:54 +0200 Subject: [ofa-general] Re: [PATCH v2 1/2] opensm: Handle trap repress on trap 144 generation In-Reply-To: <20090306181948.GA1706@comcast.net> References: <20090306181948.GA1706@comcast.net> Message-ID: <20090310153754.GG19978@sashak.voltaire.com> On 13:19 Fri 06 Mar , Hal Rosenstock wrote: > > Make trap generation sending be a transaction (expect a response) > and handle incoming trap repress (notice attribute) > ib_mad_is_response is now changed to better reflect responses > (not just R bit in MAD header) > > Signed-off-by: Hal Rosenstock Applied with discussed changes (also see below). Thanks. > @@ -624,10 +675,14 @@ __osm_sm_mad_ctrl_rcv_callback(IN osm_madw_t * p_madw, > __osm_sm_mad_ctrl_process_set(p_ctrl, p_madw); > break; > > + case IB_MAD_METHOD_TRAP_REPRESS: > + CL_ASSERT(p_req_madw == NULL); It is response, so should be CL_ASSERT(p_req_madw != NULL). Sasha From sashak at voltaire.com Tue Mar 10 08:47:25 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Mar 2009 17:47:25 +0200 Subject: [ofa-general] Re: [PATCH] libibmad/rpc.c: Handle redirection status In-Reply-To: References: <20090309135401.GA21411@comcast.net> <20090310145054.GC19978@sashak.voltaire.com> Message-ID: <20090310154725.GI19978@sashak.voltaire.com> On 09:55 Tue 10 Mar , Hal Rosenstock wrote: > > > > Basically it is about the same field. > > One bit difference. > > > Why do we need such flow? > > Strict spec compliance. Let's just mask this bit and we are done without long if/else/etc flow. > > You are sending patches which cleans {} around single expressions... > > Just wrap ERRS() macro with 'do {..} while (0)' and code will be > > cleaner. > > I'm not following what you mean by this. IOW no need to put { } braces around single expression. Sasha From sashak at voltaire.com Tue Mar 10 08:51:47 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 10 Mar 2009 17:51:47 +0200 Subject: [ofa-general] Re: [PATCHv2 2/2] libvendor/osm_vendor_mlx_dispatcher.c: Eliminate no longer needed osmv_mad_is_response In-Reply-To: <20090306182053.GB1706@comcast.net> References: <20090306182053.GB1706@comcast.net> Message-ID: <20090310155140.GJ19978@sashak.voltaire.com> On 13:20 Fri 06 Mar , Hal Rosenstock wrote: > > Use ib_mad_is_response (which now supports trap repress) > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From Ted.Kim at Sun.COM Tue Mar 10 09:45:32 2009 From: Ted.Kim at Sun.COM (Ted H. Kim) Date: Tue, 10 Mar 2009 09:45:32 -0700 Subject: [ofa-general] what uses UC in the OFED stack? Message-ID: <49B6992C.1080306@sun.com> Folks, Does anything in the OFED stack use the IB Unreliable Connected (UC) transport type? I thought maybe IPonIB-CM would because of the language in IETF RFC 4755. But the OFED-1.4 IPonIB-CM sources don't seem to use it -- unless I missed it somehow. So just to repeat the question, does anything in the OFED stack use UC? Thanks, -ted -- Ted H. Kim Sun Microsystems, Inc. ted.kim at sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From todd.rimmer at qlogic.com Tue Mar 10 09:48:36 2009 From: todd.rimmer at qlogic.com (Todd Rimmer) Date: Tue, 10 Mar 2009 11:48:36 -0500 Subject: [ofa-general] what uses UC in the OFED stack? In-Reply-To: <49B6992C.1080306@sun.com> References: <49B6992C.1080306@sun.com> Message-ID: <5AEC2602AE03EB46BFC16C6B9B200DA8134A72B406@MNEXMB2.qlogic.org> I am aware of 3rd party applications which use UC. However nothing other than perhaps test programs which ships with OFED uses UC. Why do you ask? Todd Rimmer Chief Architect QLogic Network Systems Group Voice: 610-233-4852 Fax: 610-233-4777 Todd.Rimmer at QLogic.com www.QLogic.com > -----Original Message----- > From: general-bounces at lists.openfabrics.org [mailto:general- > bounces at lists.openfabrics.org] On Behalf Of Ted H. Kim > Sent: Tuesday, March 10, 2009 12:46 PM > To: general at lists.openfabrics.org > Subject: [ofa-general] what uses UC in the OFED stack? > > Folks, > > Does anything in the OFED stack use the IB > Unreliable Connected (UC) transport type? > > I thought maybe IPonIB-CM would because > of the language in IETF RFC 4755. But > the OFED-1.4 IPonIB-CM sources don't seem > to use it -- unless I missed it somehow. > > So just to repeat the question, does > anything in the OFED stack use UC? > > Thanks, > -ted > > -- > Ted H. Kim > Sun Microsystems, Inc. ted.kim at sun.com > 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 > El Segundo, CA 90245 (310) 341-1120 FAX > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib- > general From phillipwils at gmail.com Tue Mar 10 09:54:48 2009 From: phillipwils at gmail.com (Phillip Wilson) Date: Tue, 10 Mar 2009 08:54:48 -0800 Subject: ***SPAM*** Re: [ofa-general] ***SPAM*** Re: mlx4_core 0000:c3:00.0: SW2HW_MPT failed (-16) (dmesg) In-Reply-To: <49B385C1.7090104@morey-chaisemartin.com> References: <6e4f44220903062016q8ca6d44ha22b10231ead5b3d@mail.gmail.com> <49B23680.1030508@gmail.com> <6e4f44220903080014x49591fc9idc21000b107f8771@mail.gmail.com> <49B385C1.7090104@morey-chaisemartin.com> Message-ID: <6e4f44220903100954k587ec13fobfa73cf1b801c6c0@mail.gmail.com> I worked around the mlx4_cmd_poll() device driver call timing out by implementing retries on the libverbs function call ibv_reg_mr(). So far, only one retry has been observed. I did not disable the "msi_x" driver parameter. Going forward, the PCIe systems that I will use implement only MSI for interrupts. i.e. while (retry < MAX_RETRIES) { ctx->mr = ibv_reg_mr(ctx->pd, ctx->buf, size, IBV_ACCESS_LOCAL_WRITE); if (!ctx->mr) { retry++; printf("retry[%d]\n", retry); sleep(1); } } if (retry == MAX_RETRIES) { printf("Couldn't register MR"); return NULL; } Thanks, Phillip On Sun, Mar 8, 2009 at 12:45 AM, Nicolas Morey-Chaisemartin < devel at morey-chaisemartin.com> wrote: > Phillip Wilson a écrit : > > I updated the HCA "InfiniBand: Mellanox Technologies: Unknown device > > 634a (rev a0)" to the latest firmware and issue remains. "fw_ver" is > > now 2.6.000. > > > > Any ideas on why the time out is occuring in the function? > > > > > > > > I've seen this problem couples of times. > Something on your system (probably HCA) is in a crappy state and won't > answer. > > I had the problem (with top spin HCA and old OFED stacks on IA64 system). > It was due to the driver trying to use msi_x and it failed. > Try in your modprobe.conf file to force msi=0 msi_x=0 for mlx4_core module, > it may help :) > > > Nicolas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dotanba at gmail.com Tue Mar 10 09:54:46 2009 From: dotanba at gmail.com (Dotan Barak) Date: Tue, 10 Mar 2009 18:54:46 +0200 Subject: [ofa-general] what uses UC in the OFED stack? In-Reply-To: <49B6992C.1080306@sun.com> References: <49B6992C.1080306@sun.com> Message-ID: <2f3bf9a60903100954wccb103avea008950094624a1@mail.gmail.com> Hi. I think that there isn't any ULP in the OFED stack that uses UC. Dotan On Tue, Mar 10, 2009 at 6:45 PM, Ted H. Kim wrote: > Folks, > > Does anything in the OFED stack use the IB > Unreliable Connected (UC) transport type? > > I thought maybe IPonIB-CM would because > of the language in IETF RFC 4755. But > the OFED-1.4 IPonIB-CM sources don't seem > to use it -- unless I missed it somehow. > > So just to repeat the question, does > anything in the OFED stack use UC? > > Thanks, > -ted > > -- > Ted H. Kim > Sun Microsystems, Inc.                  ted.kim at sun.com > 222 North Sepulveda Blvd., 10th Floor   (310) 341-1116 > El Segundo, CA  90245                   (310) 341-1120 FAX > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From rdreier at cisco.com Tue Mar 10 10:10:28 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 10 Mar 2009 10:10:28 -0700 Subject: [ofa-general] what uses UC in the OFED stack? In-Reply-To: <49B6992C.1080306@sun.com> (Ted H. Kim's message of "Tue, 10 Mar 2009 09:45:32 -0700") References: <49B6992C.1080306@sun.com> Message-ID: > Does anything in the OFED stack use the IB > Unreliable Connected (UC) transport type? Nothing in upstream Linux uses UC. IPoIB-CM doesn't use it because SRQ for UC was not allowed when IPoIB-CM was initially implemented, and I think it's still not fully specified (last time I checked). From sean.hefty at intel.com Tue Mar 10 10:33:46 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 10 Mar 2009 10:33:46 -0700 Subject: [ofa-general] RE: tcp/rdma port unification patch in librdmacm In-Reply-To: <49B61C5D.6060900@Voltaire.com> References: <49B17E15.4050406@opengridcomputing.com> <811553D6D34A4E6F9AB686CA2E441D35@amr.corp.intel.com> <49B61C5D.6060900@Voltaire.com> Message-ID: <3281521B4F4A415A846A9F2E8AF9C21F@amr.corp.intel.com> >As for enabling this always on librdmacm, this can't work - since it will break >IB user space app that attempts to open two listeners: I thought about that, but didn't know whether it was in use. I agree that existing apps shouldn't break. (I was thinking more along the lines of adding a new call that would make this behavior explicit, but haven't taken the time to really study the details.) >one on SID(PS_TCP, X) - for example for X=3260 and the app being the Linux stgt >target. Also Steve has mentioned earlier on this thread that kernel apps are >safe from this problem - I am not sure this is the case for Lustre. Does anyone know if a kernel patch to fix this has been accepted directly into the distros? Also, does anything keep MPI from doing exactly what we're discussing in the application as part of using the librdmacm? (Besides having all apps duplicate this.) - Sean From Ted.Kim at Sun.COM Tue Mar 10 10:36:56 2009 From: Ted.Kim at Sun.COM (Ted H. Kim) Date: Tue, 10 Mar 2009 10:36:56 -0700 Subject: [ofa-general] what uses UC in the OFED stack? In-Reply-To: References: <49B6992C.1080306@sun.com> Message-ID: <49B6A538.5000405@sun.com> Roland, Would it be fair to say that for the moment, no one seems to be really clamoring to revisit IPonIB-CM to put in UC? (i.e. RC works well enough) -ted Roland Dreier wrote: > > Does anything in the OFED stack use the IB > > Unreliable Connected (UC) transport type? > > Nothing in upstream Linux uses UC. IPoIB-CM doesn't use it because SRQ > for UC was not allowed when IPoIB-CM was initially implemented, and I > think it's still not fully specified (last time I checked). > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > -- Ted H. Kim Sun Microsystems, Inc. ted.kim at sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From rdreier at cisco.com Tue Mar 10 10:39:55 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 10 Mar 2009 10:39:55 -0700 Subject: [ofa-general] what uses UC in the OFED stack? In-Reply-To: <49B6A538.5000405@sun.com> (Ted H. Kim's message of "Tue, 10 Mar 2009 10:36:56 -0700") References: <49B6992C.1080306@sun.com> <49B6A538.5000405@sun.com> Message-ID: > Would it be fair to say that for the moment, no > one seems to be really clamoring to revisit IPonIB-CM > to put in UC? (i.e. RC works well enough) I haven't seen anyone mention it except as a theoretical question. - R. From hnrose at comcast.net Tue Mar 10 12:23:27 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 10 Mar 2009 14:23:27 -0500 Subject: [ofa-general] ***SPAM*** [PATCH 2/2] opensm: Utilize ib_gid_is_notzero routine Message-ID: <20090310192327.GB6431@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_inform.c b/opensm/opensm/osm_inform.c index f0dab30..dd3f41e 100644 --- a/opensm/opensm/osm_inform.c +++ b/opensm/opensm/osm_inform.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -108,7 +109,6 @@ __match_inf_rec(IN const cl_list_item_t * const p_list_item, IN void *context) osm_infr_t *p_infr = (osm_infr_t *) p_list_item; osm_log_t *p_log = p_infr_rec->sa->p_log; cl_status_t status = CL_NOT_FOUND; - ib_gid_t all_zero_gid; OSM_LOG_ENTER(p_log); @@ -118,11 +118,8 @@ __match_inf_rec(IN const cl_list_item_t * const p_list_item, IN void *context) goto Exit; } - memset(&all_zero_gid, 0, sizeof(ib_gid_t)); - /* if inform_info.gid is not zero, ignore lid range */ - if (memcmp(&p_infr_rec->inform_record.inform_info.gid, &all_zero_gid, - sizeof(p_infr_rec->inform_record.inform_info.gid))) { + if (ib_gid_is_notzero(&p_infr_rec->inform_record.inform_info.gid)) { if (memcmp(&p_infr->inform_record.inform_info.gid, &p_infr_rec->inform_record.inform_info.gid, sizeof(p_infr->inform_record.inform_info.gid))) { diff --git a/opensm/opensm/osm_sa_informinfo.c b/opensm/opensm/osm_sa_informinfo.c index 5863753..fb81008 100644 --- a/opensm/opensm/osm_sa_informinfo.c +++ b/opensm/opensm/osm_sa_informinfo.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -95,7 +96,6 @@ __validate_ports_access_rights(IN osm_sa_t * sa, ib_net16_t lid_range_end; ib_net16_t lid; const cl_ptr_vector_t *p_tbl; - ib_gid_t zero_gid; OSM_LOG_ENTER(sa->p_log); @@ -103,9 +103,7 @@ __validate_ports_access_rights(IN osm_sa_t * sa, p_requester_physp = osm_get_physp_by_mad_addr(sa->p_log, sa->p_subn, &p_infr_rec->report_addr); - memset(&zero_gid, 0, sizeof(zero_gid)); - if (memcmp(&(p_infr_rec->inform_record.inform_info.gid), - &zero_gid, sizeof(ib_gid_t))) { + if (ib_gid_is_notzero(&p_infr_rec->inform_record.inform_info.gid)) { /* a gid is defined */ portguid = p_infr_rec->inform_record.inform_info.gid.unicast. diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index 5de496e..ff81d25 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -2,7 +2,8 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. - * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -85,12 +86,6 @@ typedef struct osm_path_parms { boolean_t reversible; } osm_path_parms_t; -static const ib_gid_t zero_gid = { {0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00}, -}; - /********************************************************************** **********************************************************************/ static inline boolean_t @@ -777,10 +772,8 @@ __osm_pr_rcv_build_pr(IN osm_sa_t * sa, p_src_physp = p_src_port->p_physp; - if (p_dgid) { - if (memcmp(p_dgid, &zero_gid, sizeof(*p_dgid))) - is_nonzero_gid = 1; - } + if (p_dgid) + is_nonzero_gid = ib_gid_is_notzero(p_dgid); if (is_nonzero_gid) p_pr->dgid = *p_dgid; From hnrose at comcast.net Tue Mar 10 12:24:25 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 10 Mar 2009 14:24:25 -0500 Subject: [ofa-general] ***SPAM*** [PATCH][TRIVIAL] opensm/osm_trap_rcv.c: Remove extraneous comment Message-ID: <20090310192425.GC6431@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_trap_rcv.c b/opensm/opensm/osm_trap_rcv.c index 6639246..4578ebc 100644 --- a/opensm/opensm/osm_trap_rcv.c +++ b/opensm/opensm/osm_trap_rcv.c @@ -649,7 +649,6 @@ Exit: } /********************************************************************** - CURRENTLY WE ARE NOT CREATING TRAPS - SO THIS CALL IN AN ERROR **********************************************************************/ static void trap_rcv_process_response(IN osm_sm_t * sm, From hnrose at comcast.net Tue Mar 10 12:22:34 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 10 Mar 2009 14:22:34 -0500 Subject: [ofa-general] ***SPAM*** [PATCH 1/2] opensm: Add common ib_gid_is_notzero routine Message-ID: <20090310192234.GA6431@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/include/iba/ib_types.h b/opensm/include/iba/ib_types.h index e7cbb79..bfe8665 100644 --- a/opensm/include/iba/ib_types.h +++ b/opensm/include/iba/ib_types.h @@ -2338,6 +2338,40 @@ ib_gid_get_guid(IN const ib_gid_t * const p_gid) * ib_gid_t *********/ +OSM_EXPORT const ib_gid_t ib_zero_gid; + +/****f* IBA Base: Types/ib_gid_is_notzero +* NAME +* ib_gid_is_notzero +* +* DESCRIPTION +* Returns a boolean indicating whether or not the GID is zero. +* +* SYNOPSIS +*/ +static inline boolean_t OSM_API +ib_gid_is_notzero(IN const ib_gid_t * const p_gid) +{ + if (memcmp(p_gid, &ib_zero_gid, sizeof(*p_gid))) + return TRUE; + return FALSE; +} + +/* +* PARAMETERS +* p_gid +* [in] Pointer to the GID object. +* +* RETURN VALUES +* Returns TRUE if GID is not zero. +* FALSE otherwise. +* +* NOTES +* +* SEE ALSO +* ib_gid_t +*********/ + /****s* IBA Base: Types/ib_path_rec_t * NAME * ib_path_rec_t diff --git a/opensm/opensm/libopensm.map b/opensm/opensm/libopensm.map index 7cd2aba..9cfa886 100644 --- a/opensm/opensm/libopensm.map +++ b/opensm/opensm/libopensm.map @@ -18,6 +18,7 @@ OPENSM_1.5 { ib_get_sm_attr_str; ib_get_sa_attr_str; ib_get_trap_str; + ib_zero_gid; osm_dump_port_info; osm_dump_portinfo_record; osm_dump_guidinfo_record; diff --git a/opensm/opensm/osm_helper.c b/opensm/opensm/osm_helper.c index c56f5b2..b40ba0c 100644 --- a/opensm/opensm/osm_helper.c +++ b/opensm/opensm/osm_helper.c @@ -535,6 +535,8 @@ const char *ib_get_trap_str(ib_net16_t trap_num) return "Unknown"; } +const ib_gid_t ib_zero_gid = {}; + /********************************************************************** **********************************************************************/ static ib_api_status_t From hnrose at comcast.net Tue Mar 10 13:50:23 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 10 Mar 2009 15:50:23 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] infiniband-diags/perfquery.8: Update man page for PortXmit/RcvDataSL Message-ID: <20090310205023.GA1791@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/infiniband-diags/man/perfquery.8 b/infiniband-diags/man/perfquery.8 index fdbf8c4..e0743d6 100644 --- a/infiniband-diags/man/perfquery.8 +++ b/infiniband-diags/man/perfquery.8 @@ -1,22 +1,23 @@ -.TH PERFQUERY 8 "February 5, 2009" "OpenIB" "OpenIB Diagnostics" +.TH PERFQUERY 8 "March 10, 2009" "OpenIB" "OpenIB Diagnostics" .SH NAME perfquery \- query InfiniBand port counters .SH SYNOPSIS .B perfquery -[\-d(ebug)] [\-G(uid)] [-x|-extended] [-a(ll_ports)] [-l(oop_ports)] [-r(eset_after_read)] [-R(eset_only)] [\-C ca_name] [\-P ca_port] [\-t(imeout) timeout_ms] [\-V(ersion)] [\-h(elp)] [ [[port] [reset_mask]]] +[\-d(ebug)] [\-G(uid)] [\-x|\-\-extended] [\-X|\-\-xmtsl] [\-R|\-\-rcvsl] [-a(ll_ports)] [-l(oop_ports)] [-r(eset_after_read)] [-R(eset_only)] [\-C ca_name] [\-P ca_port] [\-t(imeout) timeout_ms] [\-V(ersion)] [\-h(elp)] [ [[port] [reset_mask]]] .SH DESCRIPTION .PP perfquery uses PerfMgt GMPs to obtain the PortCounters (basic performance -and error counters) or PortExtendedCounters from the PMA at the node/port -specified. Optionally shows aggregated counters for all ports of node. +and error counters), PortExtendedCounters, PortXmitDataSL, or PortRcvDataSL +from the PMA at the node/port specified. Optionally shows aggregated +counters for all ports of node. Also, optionally, reset after read, or only reset counters. -Note: In both PortCounters and PortCountersExtended, components -that represent Data (e.g. PortXmitData and PortRcvData) indicate octets -divided by 4 rather than just octets. +Note: In PortCounters, PortCountersExtended, PortXmitDataSL, and PortRcvDataSL, +components that represent Data (e.g. PortXmitData and PortRcvData) indicate +octets divided by 4 rather than just octets. Note: Inputting a port of 255 indicates an operation be performed on all ports. @@ -28,6 +29,12 @@ Note: Inputting a port of 255 indicates an operation be performed on all ports. show extended port counters rather than (basic) port counters. Note that extended port counters attribute is optional. .TP +\fB\-X\fR, \fB\-\-xmtsl\fR +show transmit data SL counter. This is an optional counter for QoS. +.TP +\fB\-S\fR, \fB\-\-rcvsl\fR +show receive data SL counter. This is an optional counter for QoS. +.TP \fB\-a\fR, \fB\-\-all_ports\fR show aggregated counters for all ports of the destination lid or reset all counters for all ports. If the destination lid From hal.rosenstock at gmail.com Tue Mar 10 14:40:10 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 10 Mar 2009 16:40:10 -0500 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] libibmad/rpc.c: Handle redirection status In-Reply-To: <20090310154725.GI19978@sashak.voltaire.com> References: <20090309135401.GA21411@comcast.net> <20090310145054.GC19978@sashak.voltaire.com> <20090310154725.GI19978@sashak.voltaire.com> Message-ID: On Tue, Mar 10, 2009 at 10:47 AM, Sasha Khapyorsky wrote: > On 09:55 Tue 10 Mar     , Hal Rosenstock wrote: >> > >> > Basically it is about the same field. >> >> One bit difference. >> >> > Why do we need such flow? >> >> Strict spec compliance. > > Let's just mask this bit and we are done without long if/else/etc flow. Sure. >> > You are sending patches which cleans {} around single expressions... >> > Just wrap ERRS() macro with 'do {..} while (0)' and code will be >> > cleaner. >> >> I'm not following what you mean by this. > > IOW no need to put { } braces around single expression. Compiler complains about ambiguous else without this. -- Hal > Sasha > From hnrose at comcast.net Tue Mar 10 15:51:14 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 10 Mar 2009 17:51:14 -0500 Subject: [ofa-general] ***SPAM*** [PATCH][TRIVIAL] libibmad/mad.h: Cosmetic formatting changes Message-ID: <20090310225114.GB3880@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h index 064cbb7..7855bad 100644 --- a/libibmad/include/infiniband/mad.h +++ b/libibmad/include/infiniband/mad.h @@ -734,13 +734,14 @@ MAD_EXPORT int mad_build_pkt(void *umad, ib_rpc_t * rpc, ib_portid_t * dport, MAD_EXPORT void madrpc_show_errors(int set); MAD_EXPORT int madrpc_set_retries(int retries); MAD_EXPORT int madrpc_set_timeout(int timeout); -MAD_EXPORT struct ibmad_port *mad_rpc_open_port(char *dev_name, int dev_port, int *mgmt_classes, - int num_classes); +MAD_EXPORT struct ibmad_port *mad_rpc_open_port(char *dev_name, int dev_port, + int *mgmt_classes, int num_classes); MAD_EXPORT void mad_rpc_close_port(struct ibmad_port *srcport); -MAD_EXPORT void *mad_rpc(const struct ibmad_port *srcport, ib_rpc_t * rpc, ib_portid_t * dport, - void *payload, void *rcvdata); -MAD_EXPORT void *mad_rpc_rmpp(const struct ibmad_port *srcport, ib_rpc_t * rpc, ib_portid_t * dport, - ib_rmpp_hdr_t * rmpp, void *data); +MAD_EXPORT void *mad_rpc(const struct ibmad_port *srcport, ib_rpc_t * rpc, + ib_portid_t * dport, void *payload, void *rcvdata); +MAD_EXPORT void *mad_rpc_rmpp(const struct ibmad_port *srcport, ib_rpc_t * rpc, + ib_portid_t * dport, ib_rmpp_hdr_t * rmpp, + void *data); MAD_EXPORT int mad_rpc_portid(struct ibmad_port *srcport); /* register.c */ @@ -749,20 +750,20 @@ MAD_EXPORT int mad_register_port_client(int port_id, int mgmt, MAD_EXPORT int mad_register_client(int mgmt, uint8_t rmpp_version) DEPRECATED; MAD_EXPORT int mad_register_server(int mgmt, uint8_t rmpp_version, - long method_mask[16 / sizeof(long)], - uint32_t class_oui) DEPRECATED; + long method_mask[16 / sizeof(long)], + uint32_t class_oui) DEPRECATED; /* register.c new interface */ MAD_EXPORT int mad_register_client_via(int mgmt, uint8_t rmpp_version, - struct ibmad_port *srcport); + struct ibmad_port *srcport); MAD_EXPORT int mad_register_server_via(int mgmt, uint8_t rmpp_version, - long method_mask[16 / sizeof(long)], - uint32_t class_oui, - struct ibmad_port *srcport); + long method_mask[16 / sizeof(long)], + uint32_t class_oui, + struct ibmad_port *srcport); MAD_EXPORT int mad_class_agent(int mgmt); /* serv.c */ MAD_EXPORT int mad_send(ib_rpc_t * rpc, ib_portid_t * dport, - ib_rmpp_hdr_t * rmpp, void *data) DEPRECATED; + ib_rmpp_hdr_t * rmpp, void *data) DEPRECATED; MAD_EXPORT void *mad_receive(void *umad, int timeout) DEPRECATED; MAD_EXPORT int mad_respond(void *umad, ib_portid_t * portid, uint32_t rstatus) @@ -770,23 +771,23 @@ MAD_EXPORT int mad_respond(void *umad, ib_portid_t * portid, uint32_t rstatus) /* serv.c new interface */ MAD_EXPORT int mad_send_via(ib_rpc_t * rpc, ib_portid_t * dport, - ib_rmpp_hdr_t * rmpp, void *data, - struct ibmad_port *srcport); + ib_rmpp_hdr_t * rmpp, void *data, + struct ibmad_port *srcport); MAD_EXPORT void *mad_receive_via(void *umad, int timeout, - struct ibmad_port *srcport); + struct ibmad_port *srcport); MAD_EXPORT int mad_respond_via(void *umad, ib_portid_t * portid, uint32_t rstatus, - struct ibmad_port *srcport); + struct ibmad_port *srcport); MAD_EXPORT void *mad_alloc(void); MAD_EXPORT void mad_free(void *umad); /* vendor.c */ MAD_EXPORT uint8_t *ib_vendor_call(void *data, ib_portid_t * portid, - ib_vendor_call_t * call) DEPRECATED; + ib_vendor_call_t * call) DEPRECATED; /* vendor.c new interface */ MAD_EXPORT uint8_t *ib_vendor_call_via(void *data, ib_portid_t * portid, - ib_vendor_call_t * call, - struct ibmad_port *srcport); + ib_vendor_call_t * call, + struct ibmad_port *srcport); static inline int mad_is_vendor_range1(int mgmt) { @@ -810,27 +811,31 @@ void madrpc_save_mad(void *madbuf, int len) DEPRECATED; /* smp.c */ MAD_EXPORT uint8_t *smp_query(void *buf, ib_portid_t * id, unsigned attrid, - unsigned mod, unsigned timeout) DEPRECATED; + unsigned mod, unsigned timeout) DEPRECATED; MAD_EXPORT uint8_t *smp_set(void *buf, ib_portid_t * id, unsigned attrid, - unsigned mod, unsigned timeout) DEPRECATED; + unsigned mod, unsigned timeout) DEPRECATED; /* smp.c new interface */ MAD_EXPORT uint8_t *smp_query_via(void *buf, ib_portid_t * id, unsigned attrid, - unsigned mod, unsigned timeout, const struct ibmad_port *srcport); -MAD_EXPORT uint8_t *smp_set_via(void *buf, ib_portid_t * id, unsigned attrid, unsigned mod, - unsigned timeout, const struct ibmad_port *srcport); + unsigned mod, unsigned timeout, + const struct ibmad_port *srcport); +MAD_EXPORT uint8_t *smp_set_via(void *buf, ib_portid_t * id, unsigned attrid, + unsigned mod, unsigned timeout, + const struct ibmad_port *srcport); /* sa.c */ uint8_t *sa_call(void *rcvbuf, ib_portid_t * portid, ib_sa_call_t * sa, unsigned timeout) DEPRECATED; -MAD_EXPORT int ib_path_query(ibmad_gid_t srcgid, ibmad_gid_t destgid, ib_portid_t * sm_id, - void *buf) DEPRECATED; +MAD_EXPORT int ib_path_query(ibmad_gid_t srcgid, ibmad_gid_t destgid, + ib_portid_t * sm_id, void *buf) DEPRECATED; /* sa.c new interface */ -MAD_EXPORT uint8_t *sa_rpc_call(const struct ibmad_port *srcport, void *rcvbuf, ib_portid_t * portid, - ib_sa_call_t * sa, unsigned timeout); -MAD_EXPORT int ib_path_query_via(const struct ibmad_port *srcport, ibmad_gid_t srcgid, - ibmad_gid_t destgid, ib_portid_t * sm_id, void *buf); +MAD_EXPORT uint8_t *sa_rpc_call(const struct ibmad_port *srcport, void *rcvbuf, + ib_portid_t * portid, ib_sa_call_t * sa, + unsigned timeout); +MAD_EXPORT int ib_path_query_via(const struct ibmad_port *srcport, + ibmad_gid_t srcgid, ibmad_gid_t destgid, + ib_portid_t * sm_id, void *buf); /* returns lid */ /* resolve.c */ @@ -848,15 +853,16 @@ MAD_EXPORT int ib_resolve_self(ib_portid_t * portid, int *portnum, /* resolve.c new interface */ MAD_EXPORT int ib_resolve_smlid_via(ib_portid_t * sm_id, int timeout, - const struct ibmad_port *srcport); + const struct ibmad_port *srcport); MAD_EXPORT int ib_resolve_guid_via(ib_portid_t * portid, uint64_t * guid, - ib_portid_t * sm_id, int timeout, - const struct ibmad_port *srcport); + ib_portid_t * sm_id, int timeout, + const struct ibmad_port *srcport); MAD_EXPORT int ib_resolve_portid_str_via(ib_portid_t * portid, char *addr_str, - enum MAD_DEST dest, ib_portid_t * sm_id, - const struct ibmad_port *srcport); -MAD_EXPORT int ib_resolve_self_via(ib_portid_t * portid, int *portnum, ibmad_gid_t * gid, - const struct ibmad_port *srcport); + enum MAD_DEST dest, ib_portid_t * sm_id, + const struct ibmad_port *srcport); +MAD_EXPORT int ib_resolve_self_via(ib_portid_t * portid, int *portnum, + ibmad_gid_t * gid, + const struct ibmad_port *srcport); /* gs.c new interface */ MAD_EXPORT uint8_t *pma_query_via(void *rcvbuf, ib_portid_t * dest, int port, From hnrose at comcast.net Tue Mar 10 15:36:53 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 10 Mar 2009 17:36:53 -0500 Subject: [ofa-general] ***SPAM*** [PATCHv2] libibmad/rpc.c: Handle redirection status Message-ID: <20090310223653.GA1479@comcast.net> Also, in mad_rpc, status should be based on management class Signed-off-by: Hal Rosenstock --- Changes since v1: Always get 16 bits of MAD status and mask when DR SMP Remove some extraneous braces diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c index 8c68cf9..965f32c 100644 --- a/libibmad/src/rpc.c +++ b/libibmad/src/rpc.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2006 Voltaire Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -170,7 +171,7 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport void *payload, void *rcvdata) { int status, len; - uint8_t sndbuf[1024], rcvbuf[1024], *mad; + uint8_t sndbuf[1024], rcvbuf[1024], *mad, mgmtclass; len = 0; memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); @@ -187,7 +188,18 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport mad = umad_get_mad(rcvbuf); - if ((status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F)) != 0) { + status = mad_get_field(mad, 0, IB_MAD_STATUS_F); + mgmtclass = mad_get_field(mad, 0, IB_MAD_MGMTCLASS_F); + if (mgmtclass == IB_SMI_DIRECT_CLASS) + status &= 0x7fff; + else if (mgmtclass != IB_SMI_CLASS) { + if (status & 2) { + ERRS("MAD redirection not supported; dport (%s)", + portid2str(dport)); + return 0; + } + } + if (status) { ERRS("MAD completed with error status 0x%x; dport (%s)", status, portid2str(dport)); return 0; @@ -227,8 +239,13 @@ void *mad_rpc_rmpp(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * mad = umad_get_mad(rcvbuf); if ((status = mad_get_field(mad, 0, IB_MAD_STATUS_F)) != 0) { - ERRS("MAD completed with error status 0x%x; dport (%s)", - status, portid2str(dport)); + if (status & 2) { + ERRS("MAD redirection not supported; dport (%s)", + portid2str(dport)); + } else { + ERRS("MAD completed with error status 0x%x; dport (%s)", + status, portid2str(dport)); + } return 0; } From weiny2 at llnl.gov Tue Mar 10 16:34:26 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Tue, 10 Mar 2009 16:34:26 -0700 Subject: [ofa-general] [PATCH 11/10] libibmad:infiniband-diags: deprecate madrpc_set_[retries|timeout] WAS: [PATCH 1/10] libibmad: Clean up "new" interface In-Reply-To: <20090307191647.GJ9673@sashak.voltaire.com> References: <20090219190525.322681b8.weiny2@llnl.gov> <20090220143402.c3b23b0a.weiny2@llnl.gov> <20090301072622.GG3936@sashak.voltaire.com> <20090302111101.bc822db8.weiny2@llnl.gov> <20090307191647.GJ9673@sashak.voltaire.com> Message-ID: <20090310163426.63d67ff2.weiny2@llnl.gov> On Sat, 7 Mar 2009 21:16:47 +0200 Sasha Khapyorsky wrote: > On 11:11 Mon 02 Mar , Ira Weiny wrote: > > > > I see this as well. Right now libibmad is designed around the "run and exit" > > diag model but we are moving it toward a library which can be used in more > > complex applications. We should push to do this once and as correct as > > possible. > > We can support both models, but we shouldn't complicate existing one, so > enforcing all existing utils to call explicitly mad_rpc_set_timeout() > is not a good idea IMO. Ok, As I said in a previous email looking deeper makes this more complicated. I think to really fix it we will have to bump the library version. This is because there are currently 2 general ways to specify a timeout, via global lib setting (madrpc_set_timeout) and via function parameter. The second way is further broken down into a via struct or a named "timeout" param. So there are kind of 3 ways to set this. In the diags they set the global and ibd_timeout to the same value and then use ibd_timeout where a "timeout" param is expected. Thereby setting the timeout the same for all calls. I will look at documenting the above as technically it is possible to change the timeout on various ports by setting the parameters appropriately on each call. I don't like this going forward but I guess it is doable without changing the interface. > BTW, this patch now breaks most of infiniband-diags tools when timeout is > not specified in command line explicitly with '-t'. The default used is > ibd_timeout, which is '0' and umad_send() fails with 'Resource > temporarily unavailable'. > Yep, missed that. Ira From boris at mellanox.com Tue Mar 10 17:20:57 2009 From: boris at mellanox.com (Boris Shpolyansky) Date: Tue, 10 Mar 2009 17:20:57 -0700 Subject: [ofa-general] ipoib performanec at smaller mtu References: <59748.128.165.248.31.1236640753.squirrel@webmail.lanl.gov> <1E3DCD1C63492545881FACB6063A57C1038A80F5@mtiexch01> <03EC693D-38BE-4947-8D7A-7C8CC3255815@lanl.gov> Message-ID: <1E3DCD1C63492545881FACB6063A57C1038A83EC@mtiexch01> Ben, What is the 8-stream BW you are seeing with IPoIB when using datagram mode and MTU=1500? Boris Shpolyansky Sr. Member of Technical Staff, Applications Mellanox Technologies Inc. 350 Oakmead Parkway, Suite 100 Sunnyvale, CA 94085 Tel.: (408) 916 0014 Fax: (408) 585 0314 Cell: (408) 834 9365 www.mellanox.com -----Original Message----- From: Ben [mailto:ben at lanl.gov] Sent: Monday, March 09, 2009 5:19 PM To: Boris Shpolyansky Cc: general at lists.openfabrics.org Subject: Re: [ofa-general] ipoib performanec at smaller mtu I have a similar setup in datagram, and don't see a big difference. Are these numbers expected? I guess I would like to understand the limitations. It seams like these numbers are a bit below what the hardware is capable of even with lro off. -Ben On Mar 9, 2009, at 6:09 PM, Boris Shpolyansky wrote: > I think for small MTU sizes you will see better performance with IPoIB > in datagram mode. > > Boris Shpolyansky > Sr. Member of Technical Staff, Applications > > Mellanox Technologies Inc. > 350 Oakmead Parkway, Suite 100 > Sunnyvale, CA 94085 > Tel.: (408) 916 0014 > Fax: (408) 585 0314 > Cell: (408) 834 9365 > www.mellanox.com > > -----Original Message----- > From: general-bounces at lists.openfabrics.org > [mailto:general-bounces at lists.openfabrics.org] On Behalf Of > Benjamin M. > McClelland > Sent: Monday, March 09, 2009 4:19 PM > To: general at lists.openfabrics.org > Subject: [ofa-general] ipoib performanec at smaller mtu > > I was curious what other people are seeing for 1500 mtu performance on > ipoib. Current tests are showing that I get about 150-175MB/s no > matter > how many streams. If I go up to higher mtu, then performance > significantly increases. Due to the storage network that this ib is > routing to, we are currently limited to 1500 mtu. > > ofed 1.4.1, connected mode, dual opteron dual core 1.8GHz, Mellanox > Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0) fw > 2.6.000, voltaire sm on switch > > lro disabled, send_queue_size=256 > > thanks, > -Ben > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general From ogerlitz at voltaire.com Wed Mar 11 00:27:23 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 11 Mar 2009 09:27:23 +0200 Subject: [ofa-general] RE: tcp/rdma port unification patch in librdmacm In-Reply-To: <3281521B4F4A415A846A9F2E8AF9C21F@amr.corp.intel.com> References: <49B17E15.4050406@opengridcomputing.com> <811553D6D34A4E6F9AB686CA2E441D35@amr.corp.intel.com> <49B61C5D.6060900@Voltaire.com> <3281521B4F4A415A846A9F2E8AF9C21F@amr.corp.intel.com> Message-ID: <49B767DB.2050700@voltaire.com> Sean Hefty wrote: > I thought about that, but didn't know whether it was in use. I agree that > existing apps shouldn't break. (I was thinking more along the lines of adding a > new call that would make this behavior explicit, but haven't taken the time to > really study the details.) > Yes, lets not break existing apps such as stgt. I'm fine with adding new call. > Does anyone know if a kernel patch to fix this has been accepted directly into > the distros? > Sorry, but I wasn't sure to follow what you mean by "fix this" ... did you refer to kernel apps that don't use different port numbers for their TCP vs RDMA listeners, or you referred to the rdma_cm patch which is not merged into mainline? > Also, does anything keep MPI from doing exactly what we're discussing in the > application as part of using the librdmacm? (Besides having all apps duplicate this > no, nothing prevents an app to open/bound a socket as long as the port is available and its ulimit allow to open the number of sockets it wants. This actually somehow brings me back to square one with regard to the actual problem, so I'll ask you guys a question as a reply to earlier post on this thread... Or. From ogerlitz at voltaire.com Wed Mar 11 00:48:37 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 11 Mar 2009 09:48:37 +0200 Subject: [ofa-general] tcp/rdma port unification patch in librdmacm In-Reply-To: <49B17E15.4050406@opengridcomputing.com> References: <49B17E15.4050406@opengridcomputing.com> Message-ID: <49B76CD5.705@voltaire.com> Steve Wise wrote: > It seems like kernel RDMA protocols are all getting IANA-assigned port > numbers that alleviates the need for TCP/RDMA port unification. > NFSRDMA and RDS are two. But user applications like mvapich2 and > other mpis often let the stack choose a local port number on-which to > bind/listen. Hi Steve, You have mentioned the listeners, but what about the ports used for the connection itself? isn't it correct that when the TOE stack accepts the rNFS connection request to port X - still a different port Y would be used for the connection itself ? but if the Linux TCP stack would have (say) iperf bind to port Y then you are again in troubles - am I missing anything here? Or. From sashak at voltaire.com Wed Mar 11 02:37:07 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 11 Mar 2009 11:37:07 +0200 Subject: [ofa-general] Re: [PATCH] libibmad/rpc.c: Handle redirection status In-Reply-To: References: <20090309135401.GA21411@comcast.net> <20090310145054.GC19978@sashak.voltaire.com> <20090310154725.GI19978@sashak.voltaire.com> Message-ID: <20090311093707.GA22085@sashak.voltaire.com> On 16:40 Tue 10 Mar , Hal Rosenstock wrote: > > > > IOW no need to put { } braces around single expression. > > Compiler complains about ambiguous else without this. I believe. This is because: #define ERRS if (iberrs || ibdebug) IBWARN So construction like: if (cond) ERRS("something"); else ... , after preprocessor it becomes: if (cond) if (iberrs || ibdebug) IBWARN("someting"); else ... , and gcc complains about ambiguous else. To avoid such sort of things I recommended in previous email to wrap ERRS() macro with do {} while (0): #define ERRS(fmt, ...) do { \ if (iberrs || ibdebug) \ IBWARN(fmt, ## __VA_ARGS__); \ } while (0) Sasha From vlad at lists.openfabrics.org Wed Mar 11 03:20:08 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Wed, 11 Mar 2009 03:20:08 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090311-0200 daily build status Message-ID: <20090311102008.966D9E60FC3@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From sashak at voltaire.com Wed Mar 11 03:29:41 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 11 Mar 2009 12:29:41 +0200 Subject: [ofa-general] [PATCH 11/10] libibmad:infiniband-diags: deprecate madrpc_set_[retries|timeout] WAS: [PATCH 1/10] libibmad: Clean up "new" interface In-Reply-To: <20090310163426.63d67ff2.weiny2@llnl.gov> References: <20090219190525.322681b8.weiny2@llnl.gov> <20090220143402.c3b23b0a.weiny2@llnl.gov> <20090301072622.GG3936@sashak.voltaire.com> <20090302111101.bc822db8.weiny2@llnl.gov> <20090307191647.GJ9673@sashak.voltaire.com> <20090310163426.63d67ff2.weiny2@llnl.gov> Message-ID: <20090311102941.GB22085@sashak.voltaire.com> On 16:34 Tue 10 Mar , Ira Weiny wrote: > > I will look at documenting the above as technically it is possible to change > the timeout on various ports by setting the parameters appropriately on each > call. I don't like this going forward but I guess it is doable without > changing the interface. And what do you say about the patch below? It doesn't change the current diags behavior and provides per port timeout/retries setup. The global parameters and rpc's fields remain to be same, and we don;t need to deprecate madrpc_set_timeout/retires(). Sasha >From 25060f5a7243a1bae6a7e7e9b2b79051ab0473d9 Mon Sep 17 00:00:00 2001 From: Sasha Khapyorsky Date: Wed, 11 Mar 2009 12:15:54 +0200 Subject: [PATCH] libibmad: per port timeout and retires setup This adds functions mad_rpc_set_timeout() and mad_rpc_set_retries() which provides possibility to specify timeout and retries for mad rpc processing per port. This doesn't change the default behavior. Signed-off-by: Sasha Khapyorsky --- libibmad/include/infiniband/mad.h | 2 ++ libibmad/src/libibmad.map | 2 ++ libibmad/src/mad_internal.h | 1 + libibmad/src/rpc.c | 37 +++++++++++++++++++++++++++---------- 4 files changed, 32 insertions(+), 10 deletions(-) diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h index 064cbb7..9b398fa 100644 --- a/libibmad/include/infiniband/mad.h +++ b/libibmad/include/infiniband/mad.h @@ -742,6 +742,8 @@ MAD_EXPORT void *mad_rpc(const struct ibmad_port *srcport, ib_rpc_t * rpc, ib_po MAD_EXPORT void *mad_rpc_rmpp(const struct ibmad_port *srcport, ib_rpc_t * rpc, ib_portid_t * dport, ib_rmpp_hdr_t * rmpp, void *data); MAD_EXPORT int mad_rpc_portid(struct ibmad_port *srcport); +MAD_EXPORT void mad_rpc_set_retries(struct ibmad_port *port, int retries); +MAD_EXPORT void mad_rpc_set_timeout(struct ibmad_port *port, int timeout); /* register.c */ MAD_EXPORT int mad_register_port_client(int port_id, int mgmt, diff --git a/libibmad/src/libibmad.map b/libibmad/src/libibmad.map index 0be7a92..10c7cd0 100644 --- a/libibmad/src/libibmad.map +++ b/libibmad/src/libibmad.map @@ -68,6 +68,8 @@ IBMAD_1.3 { mad_rpc; mad_rpc_rmpp; mad_rpc_portid; + mad_rpc_set_retries; + mad_rpc_set_timeout; madrpc; madrpc_def_timeout; madrpc_init; diff --git a/libibmad/src/mad_internal.h b/libibmad/src/mad_internal.h index e948540..24418cc 100644 --- a/libibmad/src/mad_internal.h +++ b/libibmad/src/mad_internal.h @@ -39,6 +39,7 @@ struct ibmad_port { int port_id; /* file descriptor returned by umad_open() */ int class_agents[MAX_CLASS]; /* class2agent mapper */ + int timeout, retries; }; extern struct ibmad_port *ibmp; diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c index 8c68cf9..3f3b7a2 100644 --- a/libibmad/src/rpc.c +++ b/libibmad/src/rpc.c @@ -53,7 +53,7 @@ struct ibmad_port *ibmp = &mad_port; static int iberrs; static int madrpc_retries = MAD_DEF_RETRIES; -static int def_madrpc_timeout = MAD_DEF_TIMEOUT_MS; +static int madrpc_timeout = MAD_DEF_TIMEOUT_MS; static void *save_mad; static int save_mad_len = 256; @@ -83,13 +83,23 @@ int madrpc_set_retries(int retries) int madrpc_set_timeout(int timeout) { - def_madrpc_timeout = timeout; + madrpc_timeout = timeout; return 0; } +void mad_rpc_set_retries(struct ibmad_port *port, int retries) +{ + port->retries = retries; +} + +void mad_rpc_set_timeout(struct ibmad_port *port, int timeout) +{ + port->timeout = timeout; +} + int madrpc_def_timeout(void) { - return def_madrpc_timeout; + return madrpc_timeout; } int madrpc_portid(void) @@ -104,15 +114,12 @@ int mad_rpc_portid(struct ibmad_port *srcport) static int _do_madrpc(int port_id, void *sndbuf, void *rcvbuf, int agentid, int len, - int timeout) + int timeout, int max_retries) { uint32_t trid; /* only low 32 bits */ int retries; int length, status; - if (!timeout) - timeout = def_madrpc_timeout; - if (ibdebug > 1) { IBWARN(">>> sending: len %d pktsz %zu", len, umad_size() + len); xdump(stderr, "send buf\n", sndbuf, umad_size() + len); @@ -127,7 +134,7 @@ _do_madrpc(int port_id, void *sndbuf, void *rcvbuf, int agentid, int len, trid = (uint32_t) mad_get_field64(umad_get_mad(sndbuf), 0, IB_MAD_TRID_F); - for (retries = 0; retries < madrpc_retries; retries++) { + for (retries = 0; retries < max_retries; retries++) { if (retries) { ERRS("retry %d (timeout %d ms)", retries, timeout); } @@ -171,6 +178,7 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport { int status, len; uint8_t sndbuf[1024], rcvbuf[1024], *mad; + int timeout, retries; len = 0; memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); @@ -178,9 +186,13 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport if ((len = mad_build_pkt(sndbuf, rpc, dport, 0, payload)) < 0) return 0; + timeout = rpc->timeout ? rpc->timeout : + port->timeout ? port->timeout : madrpc_timeout; + retries = port->retries ? port->retries : madrpc_retries; + if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, port->class_agents[rpc->mgtclass], - len, rpc->timeout)) < 0) { + len, timeout, retries)) < 0) { IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); return 0; } @@ -209,6 +221,7 @@ void *mad_rpc_rmpp(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * { int status, len; uint8_t sndbuf[1024], rcvbuf[1024], *mad; + int timeout, retries; memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); @@ -217,9 +230,13 @@ void *mad_rpc_rmpp(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * if ((len = mad_build_pkt(sndbuf, rpc, dport, rmpp, data)) < 0) return 0; + timeout = rpc->timeout ? rpc->timeout : + port->timeout ? port->timeout : madrpc_timeout; + retries = port->retries ? port->retries : madrpc_retries; + if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, port->class_agents[rpc->mgtclass], - len, rpc->timeout)) < 0) { + len, rpc->timeout, retries)) < 0) { IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); return 0; } -- 1.6.1.2.319.gbd9e From hnrose at comcast.net Wed Mar 11 04:30:59 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 11 Mar 2009 06:30:59 -0500 Subject: [ofa-general] ***SPAM*** [PATCHv3] libibmad/rpc.c: Handle redirection status Message-ID: <20090311113059.GB12004@comcast.net> Also, in mad_rpc, status should be based on management class Signed-off-by: Hal Rosenstock --- Changes since v2: Made ERRS into macro Changes since v1: Always get 16 bits of MAD status and mask when DR SMP Remove some extraneous braces diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c index 8c68cf9..e9a1c06 100644 --- a/libibmad/src/rpc.c +++ b/libibmad/src/rpc.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2006 Voltaire Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -59,7 +60,10 @@ static int save_mad_len = 256; #undef DEBUG #define DEBUG if (ibdebug) IBWARN -#define ERRS if (iberrs || ibdebug) IBWARN +#define ERRS(fmt, ...) do { \ + if (iberrs || ibdebug) \ + IBWARN(fmt, ## __VA_ARGS__); \ +} while (0) #define MAD_TID(mad) (*((uint64_t *)((char *)(mad) + 8))) @@ -128,9 +132,8 @@ _do_madrpc(int port_id, void *sndbuf, void *rcvbuf, int agentid, int len, (uint32_t) mad_get_field64(umad_get_mad(sndbuf), 0, IB_MAD_TRID_F); for (retries = 0; retries < madrpc_retries; retries++) { - if (retries) { + if (retries) ERRS("retry %d (timeout %d ms)", retries, timeout); - } length = len; if (umad_send(port_id, agentid, sndbuf, length, timeout, 0) < 0) { @@ -170,7 +173,7 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport void *payload, void *rcvdata) { int status, len; - uint8_t sndbuf[1024], rcvbuf[1024], *mad; + uint8_t sndbuf[1024], rcvbuf[1024], *mad, mgmtclass; len = 0; memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); @@ -187,7 +190,18 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport mad = umad_get_mad(rcvbuf); - if ((status = mad_get_field(mad, 0, IB_DRSMP_STATUS_F)) != 0) { + status = mad_get_field(mad, 0, IB_MAD_STATUS_F); + mgmtclass = mad_get_field(mad, 0, IB_MAD_MGMTCLASS_F); + if (mgmtclass == IB_SMI_DIRECT_CLASS) + status &= 0x7fff; + else if (mgmtclass != IB_SMI_CLASS) { + if (status & 2) { + ERRS("MAD redirection not supported; dport (%s)", + portid2str(dport)); + return 0; + } + } + if (status) { ERRS("MAD completed with error status 0x%x; dport (%s)", status, portid2str(dport)); return 0; @@ -227,8 +241,12 @@ void *mad_rpc_rmpp(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * mad = umad_get_mad(rcvbuf); if ((status = mad_get_field(mad, 0, IB_MAD_STATUS_F)) != 0) { - ERRS("MAD completed with error status 0x%x; dport (%s)", - status, portid2str(dport)); + if (status & 2) + ERRS("MAD redirection not supported; dport (%s)", + portid2str(dport)); + else + ERRS("MAD completed with error status 0x%x; dport (%s)", + status, portid2str(dport)); return 0; } From hnrose at comcast.net Wed Mar 11 04:30:12 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 11 Mar 2009 06:30:12 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] infiniband-diags/perfquery.c: Fix some memory leaks on exit Message-ID: <20090311113012.GA12004@comcast.net> Signed-off-by: Hal Rosenstock --- Resend of patch sent yesterday as I fat fingered the list address diff --git a/infiniband-diags/src/perfquery.c b/infiniband-diags/src/perfquery.c index 39b3135..d35e995 100644 --- a/infiniband-diags/src/perfquery.c +++ b/infiniband-diags/src/perfquery.c @@ -473,12 +473,12 @@ int main(int argc, char **argv) if (xmt_sl) { xmt_sl_query(&portid, port, mask); - exit(0); + goto done; } if (rcv_sl) { rcv_sl_query(&portid, port, mask); - exit(0); + goto done; } if (all_ports_loop || (loop_ports && (all_ports || port == ALL_PORTS))) { @@ -519,10 +519,8 @@ int main(int argc, char **argv) else dump_perfcounters(extended, ibd_timeout, cap_mask, &portid, port, 0); - if (!reset) { - mad_rpc_close_port(srcport); - exit(0); - } + if (!reset) + goto done; do_reset: @@ -536,6 +534,7 @@ do_reset: else reset_counters(extended, ibd_timeout, mask, &portid, port); +done: mad_rpc_close_port(srcport); exit(0); } From sashak at voltaire.com Wed Mar 11 03:54:42 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 11 Mar 2009 12:54:42 +0200 Subject: [ofa-general] [PATCH] libibmad: add mad_rpc_class_agent() call In-Reply-To: <20090219190525.322681b8.weiny2@llnl.gov> References: <20090219190525.322681b8.weiny2@llnl.gov> Message-ID: <20090311105442.GD22085@sashak.voltaire.com> Add new thread safe mad_rpc_class_agent() class which gets port as parameter. Deprecate old mad_class_agent() and convert its usage in ibsysstat. Signed-off-by: Sasha Khapyorsky --- infiniband-diags/src/ibsysstat.c | 7 ++++--- libibmad/include/infiniband/mad.h | 3 ++- libibmad/src/libibmad.map | 1 + libibmad/src/rpc.c | 9 ++++++++- 4 files changed, 15 insertions(+), 5 deletions(-) diff --git a/infiniband-diags/src/ibsysstat.c b/infiniband-diags/src/ibsysstat.c index 8f879ae..03c0515 100644 --- a/infiniband-diags/src/ibsysstat.c +++ b/infiniband-diags/src/ibsysstat.c @@ -103,8 +103,9 @@ static int server_respond(void *umad, int size) if (ibdebug > 1) xdump(stderr, "mad respond pkt\n", mad, IB_MAD_SIZE); - if (umad_send(mad_rpc_portid(srcport), mad_class_agent(rpc.mgtclass), umad, - size, rpc.timeout, 0) < 0) { + if (umad_send(mad_rpc_portid(srcport), + mad_rpc_class_agent(srcport, rpc.mgtclass), umad, size, + rpc.timeout, 0) < 0) { DEBUG("send failed; %m"); return -1; } @@ -233,7 +234,7 @@ static char *ibsystat(ib_portid_t *portid, int attr) IBPANIC("cannot build packet."); fd = mad_rpc_portid(srcport); - agent = mad_class_agent(rpc.mgtclass); + agent = mad_rpc_class_agent(srcport, rpc.mgtclass); timeout = ibd_timeout ? ibd_timeout : MAD_DEF_TIMEOUT_MS; if (umad_send(fd, agent, buf, len, timeout, 0) < 0) diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h index 9b398fa..6611d9e 100644 --- a/libibmad/include/infiniband/mad.h +++ b/libibmad/include/infiniband/mad.h @@ -744,6 +744,7 @@ MAD_EXPORT void *mad_rpc_rmpp(const struct ibmad_port *srcport, ib_rpc_t * rpc, MAD_EXPORT int mad_rpc_portid(struct ibmad_port *srcport); MAD_EXPORT void mad_rpc_set_retries(struct ibmad_port *port, int retries); MAD_EXPORT void mad_rpc_set_timeout(struct ibmad_port *port, int timeout); +MAD_EXPORT int mad_rpc_class_agent(struct ibmad_port *srcport, int cls); /* register.c */ MAD_EXPORT int mad_register_port_client(int port_id, int mgmt, @@ -760,7 +761,7 @@ MAD_EXPORT int mad_register_server_via(int mgmt, uint8_t rmpp_version, long method_mask[16 / sizeof(long)], uint32_t class_oui, struct ibmad_port *srcport); -MAD_EXPORT int mad_class_agent(int mgmt); +MAD_EXPORT int mad_class_agent(int mgmt) DEPRECATED; /* serv.c */ MAD_EXPORT int mad_send(ib_rpc_t * rpc, ib_portid_t * dport, diff --git a/libibmad/src/libibmad.map b/libibmad/src/libibmad.map index 10c7cd0..b7c2031 100644 --- a/libibmad/src/libibmad.map +++ b/libibmad/src/libibmad.map @@ -68,6 +68,7 @@ IBMAD_1.3 { mad_rpc; mad_rpc_rmpp; mad_rpc_portid; + mad_rpc_class_agent; mad_rpc_set_retries; mad_rpc_set_timeout; madrpc; diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c index 3f3b7a2..233874b 100644 --- a/libibmad/src/rpc.c +++ b/libibmad/src/rpc.c @@ -109,7 +109,14 @@ int madrpc_portid(void) int mad_rpc_portid(struct ibmad_port *srcport) { - return (srcport->port_id); + return srcport->port_id; +} + +int mad_rpc_class_agent(struct ibmad_port *port, int class) +{ + if (class < 1 || class > MAX_CLASS) + return -1; + return port->class_agents[class]; } static int -- 1.6.1.2.319.gbd9e From sashak at voltaire.com Wed Mar 11 03:55:59 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 11 Mar 2009 12:55:59 +0200 Subject: [ofa-general] [PATCH] libibmad: deprecate old API calls In-Reply-To: <20090311105442.GD22085@sashak.voltaire.com> References: <20090219190525.322681b8.weiny2@llnl.gov> <20090311105442.GD22085@sashak.voltaire.com> Message-ID: <20090311105559.GE22085@sashak.voltaire.com> Now when libibmad and infiniband-diags interfaces are cleaned we can deprecate an old not thread safe libibmad API calls. Signed-off-by: Sasha Khapyorsky --- libibmad/include/infiniband/mad_osd.h | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/libibmad/include/infiniband/mad_osd.h b/libibmad/include/infiniband/mad_osd.h index ff43fa7..6315514 100644 --- a/libibmad/include/infiniband/mad_osd.h +++ b/libibmad/include/infiniband/mad_osd.h @@ -44,6 +44,6 @@ #include #define MAD_EXPORT -#define DEPRECATED /* __attribute__ ((deprecated)) */ +#define DEPRECATED __attribute__ ((deprecated)) #endif /* _MAD_OSD_H_ */ -- 1.6.1.2.319.gbd9e From hnrose at comcast.net Wed Mar 11 04:59:46 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 11 Mar 2009 06:59:46 -0500 Subject: [ofa-general] ***SPAM*** [PATCH 2/2] opensm/osm_console.c: Add dump and clear redir perfmgr command support Message-ID: <20090311115946.GB21929@comcast.net> Also, cosmetic commentary change Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_console.c b/opensm/opensm/osm_console.c index 97be931..a2160f9 100644 --- a/opensm/opensm/osm_console.c +++ b/opensm/opensm/osm_console.c @@ -211,7 +211,7 @@ static void help_dump_conf(FILE *out, int detail) static void help_perfmgr(FILE * out, int detail) { fprintf(out, - "perfmgr [enable|disable|clear_counters|dump_counters|print_counters|sweep_time[seconds]]\n"); + "perfmgr [enable|disable|clear_counters|dump_counters|print_counters|dump_redir|clear_redir|sweep_time[seconds]]\n"); if (detail) { fprintf(out, "perfmgr -- print the performance manager state\n"); @@ -225,6 +225,10 @@ static void help_perfmgr(FILE * out, int detail) " [dump_counters [mach]] -- dump the counters (optionally in [mach]ine readable format)\n"); fprintf(out, " [print_counters ] -- print the counters for the specified node\n"); + fprintf(out, + " [dump_redir []] -- dump the redirection table\n"); + fprintf(out, + " [clear_redir []] -- clear the redirection table\n"); } } #endif /* ENABLE_OSM_PERF_MGR */ @@ -1033,8 +1037,8 @@ static void lidbalance_check(osm_opensm_t * p_osm, port_max_count = port_count[i]; } - /* Output if this CA/router is being forwarded an unbalanced number of - * times to a destination. + /* Output if this CA/router is being forwarded an unbalanced + * number of times to a destination. */ if ((rem_max_count - rem_min_count) > 1) { fprintf(out, @@ -1135,6 +1139,152 @@ static void dump_conf_parse(char **p_last, osm_opensm_t * p_osm, FILE * out) } #ifdef ENABLE_OSM_PERF_MGR +static __monitored_node_t *find_node_by_name(osm_opensm_t * p_osm, + char *nodename) +{ + cl_map_item_t *item; + __monitored_node_t *node; + + item = cl_qmap_head(&p_osm->perfmgr.monitored_map); + while (item != cl_qmap_end(&p_osm->perfmgr.monitored_map)) { + node = (__monitored_node_t *)item; + if (strcmp(node->name, nodename) == 0) + return node; + item = cl_qmap_next(item); + } + + return NULL; +} + +static __monitored_node_t *find_node_by_guid(osm_opensm_t * p_osm, + uint64_t guid) +{ + cl_map_item_t *node; + + node = cl_qmap_get(&p_osm->perfmgr.monitored_map, guid); + if (node != cl_qmap_end(&p_osm->perfmgr.monitored_map)) + return (__monitored_node_t *)node; + + return NULL; +} + +static void dump_redir_entry(__monitored_node_t *p_mon_node, FILE * out) +{ + int port, redir; + + /* only display monitored nodes with redirection info */ + redir = 0; + for (port = (p_mon_node->esp0) ? 0 : 1; + port < p_mon_node->redir_tbl_size; port++) { + if (p_mon_node->redir_port[port].redirection) { + if (!redir) { + fprintf(out, " Node GUID ESP0 Name\n"); + fprintf(out, " --------- ---- ----\n"); + fprintf(out, " 0x%" PRIx64 " %d %s\n", + p_mon_node->guid, p_mon_node->esp0, + p_mon_node->name); + fprintf(out, "\n Port Invalid LIDs PKey QP PKey Index\n"); + fprintf(out, " ---- ------- ---- ---- -- ----------\n"); + redir = 1; + } + /* per port redir_t */ + fprintf(out, " %d %d %u->%u 0x%x 0x%x %d\n", + port, p_mon_node->redir_port[port].invalid, + cl_ntoh16(p_mon_node->redir_port[port].orig_lid), + cl_ntoh16(p_mon_node->redir_port[port].redir_lid), + cl_ntoh16(p_mon_node->redir_port[port].redir_pkey), + cl_ntoh32(p_mon_node->redir_port[port].redir_qp), + p_mon_node->redir_port[port].redir_pkey_ix); + } + } + if (redir) + fprintf(out, "\n"); +} + +static void dump_redir(osm_opensm_t * p_osm, char *nodename, FILE * out) +{ + __monitored_node_t *p_mon_node; + uint64_t guid; + + if (!p_osm->subn.opt.perfmgr_redir) + fprintf(out, "Perfmgr redirection not enabled\n"); + + fprintf(out, "\nRedirection Table\n"); + fprintf(out, "-----------------\n"); + cl_plock_acquire(p_osm->perfmgr.lock); + if (nodename) { + guid = strtoull(nodename, NULL, 0); + if (guid == 0 && errno) + p_mon_node = find_node_by_name(p_osm, nodename); + else + p_mon_node = find_node_by_guid(p_osm, guid); + if (p_mon_node) + dump_redir_entry(p_mon_node, out); + else { + if (guid == 0 && errno) + fprintf(out, "Node %s not found...\n", nodename); + else + fprintf(out, "Node 0x%" PRIx64 " not found...\n", guid); + } + } else { + p_mon_node = (__monitored_node_t *) cl_qmap_head(&p_osm->perfmgr.monitored_map); + while (p_mon_node != (__monitored_node_t *) cl_qmap_end(&p_osm->perfmgr.monitored_map)) { + dump_redir_entry(p_mon_node, out); + p_mon_node = (__monitored_node_t *) cl_qmap_next((const cl_map_item_t *)p_mon_node); + } + } + cl_plock_release(p_osm->perfmgr.lock); +} + +static void clear_redir_entry(__monitored_node_t *p_mon_node) +{ + int port; + ib_net16_t orig_lid; + + for (port = (p_mon_node->esp0) ? 0 : 1; + port < p_mon_node->redir_tbl_size; port++) { + if (p_mon_node->redir_port[port].redirection) { + orig_lid = p_mon_node->redir_port[port].orig_lid; + memset(&p_mon_node->redir_port[port], 0, + sizeof(redir_t)); + p_mon_node->redir_port[port].orig_lid = orig_lid; + } + } +} + +static void clear_redir(osm_opensm_t * p_osm, char *nodename, FILE * out) +{ + __monitored_node_t *p_mon_node; + uint64_t guid; + + if (!p_osm->subn.opt.perfmgr_redir) + fprintf(out, "Perfmgr redirection not enabled\n"); + + cl_plock_acquire(p_osm->perfmgr.lock); + if (nodename) { + guid = strtoull(nodename, NULL, 0); + if (guid == 0 && errno) + p_mon_node = find_node_by_name(p_osm, nodename); + else + p_mon_node = find_node_by_guid(p_osm, guid); + if (p_mon_node) + clear_redir_entry(p_mon_node); + else { + if (guid == 0 && errno) + fprintf(out, "Node %s not found...\n", nodename); + else + fprintf(out, "Node 0x%" PRIx64 " not found...\n", guid); + } + } else { + p_mon_node = (__monitored_node_t *) cl_qmap_head(&p_osm->perfmgr.monitored_map); + while (p_mon_node != (__monitored_node_t *) cl_qmap_end(&p_osm->perfmgr.monitored_map)) { + clear_redir_entry(p_mon_node); + p_mon_node = (__monitored_node_t *) cl_qmap_next((const cl_map_item_t *)p_mon_node); + } + } + cl_plock_release(p_osm->perfmgr.lock); +} + static void perfmgr_parse(char **p_last, osm_opensm_t * p_osm, FILE * out) { char *p_cmd; @@ -1167,6 +1317,12 @@ static void perfmgr_parse(char **p_last, osm_opensm_t * p_osm, FILE * out) fprintf(out, "print_counters requires a node name or node GUID to be specified\n"); } + } else if (strcmp(p_cmd, "dump_redir") == 0) { + p_cmd = name_token(p_last); + dump_redir(p_osm, p_cmd, out); + } else if (strcmp(p_cmd, "clear_redir") == 0) { + p_cmd = name_token(p_last); + clear_redir(p_osm, p_cmd, out); } else if (strcmp(p_cmd, "sweep_time") == 0) { p_cmd = next_token(p_last); if (p_cmd) { @@ -1403,9 +1559,8 @@ int osm_console(osm_opensm_t * p_osm) if (p_oct->out) { loop_command.loop_function(p_osm, p_oct->out); fflush(p_oct->out); - } else { + } else loop_command.on = 0; - } } if (poll(fds, nfds, 1000) <= 0) From hnrose at comcast.net Wed Mar 11 04:58:01 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 11 Mar 2009 06:58:01 -0500 Subject: [ofa-general] ***SPAM*** [PATCH 1/2] opensm/PerfMgr: Better redirection support Message-ID: <20090311115801.GA21929@comcast.net> Handle PKey and QPN redirection information GID redirection handling remains Signed-off-by: Hal Rosenstock --- This patch relies on patch sent yesterday adding ib_gid_is_notzero diff --git a/opensm/include/opensm/osm_perfmgr.h b/opensm/include/opensm/osm_perfmgr.h index 45dec54..651ea80 100644 --- a/opensm/include/opensm/osm_perfmgr.h +++ b/opensm/include/opensm/osm_perfmgr.h @@ -92,8 +92,14 @@ typedef enum { /* Redirection information */ typedef struct redir { + boolean_t redirection; + boolean_t invalid; + ib_gid_t redir_gid; ib_net16_t redir_lid; + ib_net16_t redir_pkey; ib_net32_t redir_qp; + uint16_t redir_pkey_ix; + ib_net16_t orig_lid; } redir_t; /* Node to store information about which nodes we are monitoring */ @@ -134,6 +140,7 @@ typedef struct osm_perfmgr { uint32_t max_outstanding_queries; cl_qmap_t monitored_map; /* map the nodes we are tracking */ __monitored_node_t *remove_list; + int16_t local_port; } osm_perfmgr_t; /* * FIELDS diff --git a/opensm/opensm/osm_perfmgr.c b/opensm/opensm/osm_perfmgr.c index 4a6f65c..7f8c8a4 100644 --- a/opensm/opensm/osm_perfmgr.c +++ b/opensm/opensm/osm_perfmgr.c @@ -47,7 +47,6 @@ #endif /* HAVE_CONFIG_H */ #ifdef ENABLE_OSM_PERF_MGR - #include #include #include @@ -67,6 +66,8 @@ #include #define OSM_PERFMGR_INITIAL_TID_VALUE 0xcafe +#define MAX_LOCAL_IBPORTS 64 +#define MAX_LOCAL_PKEYS 256 #if ENABLE_OSM_PERF_MGR_PROFILE struct { @@ -118,8 +119,6 @@ static inline void diff_time(struct timeval *before, } #endif -extern int wait_for_pending_transactions(osm_stats_t * stats); - /********************************************************************** * Internal helper functions. **********************************************************************/ @@ -203,6 +202,7 @@ osm_perfmgr_mad_send_err_callback(void *bind_context, osm_madw_t * p_madw) uint8_t port = context->perfmgr_context.port; cl_map_item_t *p_node; __monitored_node_t *p_mon_node; + ib_net16_t orig_lid; OSM_LOG_ENTER(pm->log); @@ -233,9 +233,10 @@ osm_perfmgr_mad_send_err_callback(void *bind_context, osm_madw_t * p_madw) p_mon_node->guid, p_mon_node->redir_tbl_size); goto Exit; } - /* Clear redirection info */ - p_mon_node->redir_port[port].redir_lid = 0; - p_mon_node->redir_port[port].redir_qp = 0; + /* Clear redirection info for this port except orig_lid */ + orig_lid = p_mon_node->redir_port[port].orig_lid; + memset(&p_mon_node->redir_port[port], 0, sizeof(redir_t)); + p_mon_node->redir_port[port].orig_lid = orig_lid; cl_plock_release(pm->lock); } @@ -292,6 +293,40 @@ osm_perfmgr_bind(osm_perfmgr_t * const pm, const ib_net64_t port_guid) goto Exit; } + /* if redirection enabled, determine local port from port GUID */ + if (pm->subn->opt.perfmgr_redir) { + ib_port_attr_t attr_array[MAX_LOCAL_IBPORTS]; + uint32_t num_ports = MAX_LOCAL_IBPORTS; + int i; + + for (i = 0; i < num_ports; i++) { + attr_array[i].num_pkeys = 0; + attr_array[i].p_pkey_table = NULL; + } + + /* call transport layer for a list of local port GUIDs */ + status = osm_vendor_get_all_port_attr(pm->subn->p_osm->p_vendor, + attr_array, &num_ports); + if (status != IB_SUCCESS) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C1C: osm_vendor_get_all_port_attr status 0x%x\n", + status); + goto Exit; + } + if (num_ports == 0) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C1D: No local ports detected!\n"); + goto Exit; + } + + for (i = 0; i < num_ports; i++) { + if (port_guid == attr_array[i].port_guid) { + pm->local_port = attr_array[i].port_num; + break; + } + } + } + Exit: OSM_LOG_EXIT(pm->log); return (status); @@ -321,25 +356,16 @@ static ib_net32_t get_qp(__monitored_node_t * mon_node, uint8_t port) if (mon_node && mon_node->redir_tbl_size && port < mon_node->redir_tbl_size && - mon_node->redir_port[port].redir_lid && + mon_node->redir_port[port].redirection && mon_node->redir_port[port].redir_qp) qp = mon_node->redir_port[port].redir_qp; return qp; } -/********************************************************************** - * Given a node, a port, and an optional monitored node, - * return the appropriate lid to query that port - **********************************************************************/ static ib_net16_t -get_lid(osm_node_t * p_node, uint8_t port, __monitored_node_t * mon_node) +get_base_lid(osm_node_t * p_node, uint8_t port) { - if (mon_node && mon_node->redir_tbl_size && - port < mon_node->redir_tbl_size && - mon_node->redir_port[port].redir_lid) - return mon_node->redir_port[port].redir_lid; - switch (p_node->node_info.node_type) { case IB_NODE_TYPE_CA: case IB_NODE_TYPE_ROUTER: @@ -352,12 +378,27 @@ get_lid(osm_node_t * p_node, uint8_t port, __monitored_node_t * mon_node) } /********************************************************************** + * Given a node, a port, and an optional monitored node, + * return the lid appropriate to query that port + **********************************************************************/ +static ib_net16_t +get_lid(osm_node_t * p_node, uint8_t port, __monitored_node_t * mon_node) +{ + if (mon_node && mon_node->redir_tbl_size && + port < mon_node->redir_tbl_size && + mon_node->redir_port[port].redir_lid) + return mon_node->redir_port[port].redir_lid; + + return get_base_lid(p_node, port); +} + +/********************************************************************** * Form and send the Port Counters MAD for a single port. **********************************************************************/ static ib_api_status_t osm_perfmgr_send_pc_mad(osm_perfmgr_t * perfmgr, ib_net16_t dest_lid, - ib_net32_t dest_qp, uint8_t port, uint8_t mad_method, - osm_madw_context_t * const p_context) + ib_net32_t dest_qp, uint16_t pkey_ix, uint8_t port, + uint8_t mad_method, osm_madw_context_t * const p_context) { ib_api_status_t status = IB_SUCCESS; ib_port_counters_t *port_counter = NULL; @@ -396,8 +437,7 @@ osm_perfmgr_send_pc_mad(osm_perfmgr_t * perfmgr, ib_net16_t dest_lid, p_madw->mad_addr.addr_type.gsi.remote_qp = dest_qp; p_madw->mad_addr.addr_type.gsi.remote_qkey = cl_hton32(IB_QP1_WELL_KNOWN_Q_KEY); - /* FIXME what about other partitions */ - p_madw->mad_addr.addr_type.gsi.pkey_ix = 0; + p_madw->mad_addr.addr_type.gsi.pkey_ix = pkey_ix; p_madw->mad_addr.addr_type.gsi.service_level = 0; p_madw->mad_addr.addr_type.gsi.global_route = FALSE; p_madw->resp_expected = TRUE; @@ -432,28 +472,32 @@ static void __collect_guids(cl_map_item_t * const p_map_item, void *context) uint64_t node_guid = cl_ntoh64(node->node_info.node_guid); osm_perfmgr_t *pm = (osm_perfmgr_t *) context; __monitored_node_t *mon_node = NULL; - uint32_t size; + uint32_t num_ports; + int port; OSM_LOG_ENTER(pm->log); if (cl_qmap_get(&pm->monitored_map, node_guid) == cl_qmap_end(&pm->monitored_map)) { /* if not already in our map add it */ - size = osm_node_get_num_physp(node); - mon_node = malloc(sizeof(*mon_node) + sizeof(redir_t) * size); + num_ports = osm_node_get_num_physp(node); + mon_node = malloc(sizeof(*mon_node) + sizeof(redir_t) * num_ports); if (!mon_node) { OSM_LOG(pm->log, OSM_LOG_ERROR, "PerfMgr: ERR 4C06: " "malloc failed: not handling node %s" "(GUID 0x%" PRIx64 ")\n", node->print_desc, node_guid); goto Exit; } - memset(mon_node, 0, sizeof(*mon_node) + sizeof(redir_t) * size); + memset(mon_node, 0, sizeof(*mon_node) + sizeof(redir_t) * num_ports); mon_node->guid = node_guid; mon_node->name = strdup(node->print_desc); - mon_node->redir_tbl_size = size; + mon_node->redir_tbl_size = num_ports; /* check for enhanced switch port 0 */ mon_node->esp0 = (node->sw && ib_switch_info_is_enhanced_port0(&node->sw->switch_info)); + for (port = (mon_node->esp0) ? 0 : 1; port < num_ports; port++) + mon_node->redir_port[port].orig_lid = get_base_lid(node, port); + cl_qmap_insert(&(pm->monitored_map), node_guid, (cl_map_item_t *) mon_node); } @@ -511,6 +555,10 @@ __osm_perfmgr_query_counters(cl_map_item_t * const p_map_item, void *context) if (!osm_node_get_physp_ptr(node, port)) continue; + if (mon_node->redir_port[port].redirection && + mon_node->redir_port[port].invalid) + continue; + lid = get_lid(node, port, mon_node); if (lid == 0) { OSM_LOG(pm->log, OSM_LOG_DEBUG, "WARN: node 0x%" PRIx64 @@ -532,8 +580,10 @@ __osm_perfmgr_query_counters(cl_map_item_t * const p_map_item, void *context) PRIx64 " port %d (lid %u) (%s)\n", node_guid, port, cl_ntoh16(lid), node->print_desc); status = - osm_perfmgr_send_pc_mad(pm, lid, remote_qp, port, - IB_MAD_METHOD_GET, &mad_context); + osm_perfmgr_send_pc_mad(pm, lid, remote_qp, + mon_node->redir_port[port].redir_pkey_ix, + port, IB_MAD_METHOD_GET, + &mad_context); if (status != IB_SUCCESS) OSM_LOG(pm->log, OSM_LOG_ERROR, "ERR 4C09: " "Failed to issue port counter query for node 0x%" @@ -550,6 +600,7 @@ Exit: * Discovery stuff. * Basically this code should not be here, but merged with main OpenSM **********************************************************************/ +extern int wait_for_pending_transactions(osm_stats_t * stats); extern void osm_drop_mgr_process(IN osm_sm_t *sm); static int sweep_hop_1(osm_sm_t * sm) @@ -980,6 +1031,10 @@ osm_perfmgr_check_overflow(osm_perfmgr_t * pm, __monitored_node_t *mon_node, osm_node_t *p_node = NULL; ib_net16_t lid = 0; + if (mon_node->redir_port[port].redirection && + mon_node->redir_port[port].invalid) + goto Exit; + osm_log(pm->log, OSM_LOG_VERBOSE, "PerfMgr: Counter overflow: %s (0x%" PRIx64 ") port %d; clearing counters\n", @@ -1004,8 +1059,10 @@ osm_perfmgr_check_overflow(osm_perfmgr_t * pm, __monitored_node_t *mon_node, mad_context.perfmgr_context.mad_method = IB_MAD_METHOD_SET; /* clear port counters */ status = - osm_perfmgr_send_pc_mad(pm, lid, remote_qp, port, - IB_MAD_METHOD_SET, &mad_context); + osm_perfmgr_send_pc_mad(pm, lid, remote_qp, + mon_node->redir_port[port].redir_pkey_ix, + port, IB_MAD_METHOD_SET, + &mad_context); if (status != IB_SUCCESS) OSM_LOG(pm->log, OSM_LOG_ERROR, "PerfMgr: ERR 4C11: " "Failed to send clear counters MAD for %s (0x%" @@ -1063,6 +1120,73 @@ osm_perfmgr_log_events(osm_perfmgr_t * pm, __monitored_node_t *mon_node, uint8_t time_diff, mon_node->name, mon_node->guid, port); } +static boolean_t validate_redir_pkey(osm_perfmgr_t *pm, ib_net16_t pkey, + uint16_t *pkey_ix) +{ + ib_port_attr_t attr_array[MAX_LOCAL_IBPORTS]; + uint32_t num_ports = MAX_LOCAL_IBPORTS; + ib_api_status_t status; + boolean_t sts = FALSE; + uint16_t i = 0; + + OSM_LOG_ENTER(pm->log); + + for (i = 0; i < num_ports; i++) { + attr_array[i].num_pkeys = 0; + attr_array[i].p_pkey_table = NULL; + } + + /* If local port couldn't be determined previously */ + if (pm->local_port == -1) + goto not_found; + + attr_array[pm->local_port].num_pkeys = MAX_LOCAL_PKEYS; + attr_array[pm->local_port].p_pkey_table = + malloc(MAX_LOCAL_PKEYS * sizeof(ib_net16_t)); + if (!attr_array[pm->local_port].p_pkey_table) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C20: No memory for port %d pkey table\n", + pm->local_port); + goto not_found; + } + + /* call the transport layer for a list of local port pkeys */ + status = osm_vendor_get_all_port_attr(pm->subn->p_osm->p_vendor, + attr_array, &num_ports); + if (status != IB_SUCCESS) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C1E: osm_vendor_get_all_port_attr status 0x%x\n", + status); + goto not_found; + } + if (num_ports == 0 || pm->local_port > num_ports) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C1F: No local ports detected or local port out of range!\n"); + goto not_found; + } + ib_net16_t *pkey_table = attr_array[pm->local_port].p_pkey_table; + for (i = 0; i < attr_array[pm->local_port].num_pkeys; i++) + if (pkey_table[i] == pkey) + break; + if (i == attr_array[pm->local_port].num_pkeys) { + i = 0; + goto not_found; + } + free(attr_array[pm->local_port].p_pkey_table); + sts = TRUE; + goto Exit; + +not_found: + if (attr_array[pm->local_port].p_pkey_table) + free(attr_array[pm->local_port].p_pkey_table); + sts = FALSE; +Exit: + if (pkey_ix) + *pkey_ix = i; + OSM_LOG_EXIT(pm->log); + return sts; +} + /********************************************************************** * The dispatcher uses a thread pool which will call this function when * we have a thread available to process our mad received from the wire. @@ -1082,6 +1206,8 @@ static void osm_pc_rcv_process(void *context, void *data) perfmgr_db_data_cnt_reading_t data_reading; cl_map_item_t *p_node; __monitored_node_t *p_mon_node; + uint16_t pkey_ix; + boolean_t invalid = FALSE; OSM_LOG_ENTER(pm->log); @@ -1105,7 +1231,8 @@ static void osm_pc_rcv_process(void *context, void *data) p_mad->attr_id == IB_MAD_ATTR_CLASS_PORT_INFO); /* Response could also be redirection (IBM eHCA PMA does this) */ - if (p_mad->attr_id == IB_MAD_ATTR_CLASS_PORT_INFO) { + if (p_mad->status & IB_MAD_STATUS_REDIRECT && + p_mad->attr_id == IB_MAD_ATTR_CLASS_PORT_INFO) { char gid_str[INET6_ADDRSTRLEN]; ib_class_port_info_t *cpi = (ib_class_port_info_t *) & @@ -1119,17 +1246,48 @@ static void osm_pc_rcv_process(void *context, void *data) sizeof gid_str), cl_ntoh32(cpi->redir_qp)); - /* LID or GID redirection ? */ - /* For GID redirection, need to get PathRecord from SA */ + /* valid redirection ? */ if (cpi->redir_lid == 0) { - OSM_LOG(pm->log, OSM_LOG_VERBOSE, - "GID redirection not currently implemented!\n"); - goto Exit; + if (!ib_gid_is_notzero(&cpi->redir_gid)) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C17: Invalid redirection " + "(both redirect LID and GID are zero)\n"); + invalid = TRUE; + } + } + if (cpi->redir_qp == 0) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C18: Invalid RedirectQP\n"); + invalid = TRUE; + } + if (cpi->redir_pkey == 0) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C19: Invalid RedirectP_Key\n"); + invalid = TRUE; + } + if (cpi->redir_qkey != IB_QP1_WELL_KNOWN_Q_KEY) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C1A: Invalid RedirectQ_Key\n"); + invalid = TRUE; + } + + if (!validate_redir_pkey(pm, cpi->redir_pkey, &pkey_ix)) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C1B: Index for Pkey 0x%x not found\n", + cl_ntoh16(cpi->redir_pkey)); + invalid = TRUE; } if (!pm->subn->opt.perfmgr_redir) { - OSM_LOG(pm->log, OSM_LOG_ERROR, "ERR 4C16: " - "redirection requested but disabled\n"); + OSM_LOG(pm->log, OSM_LOG_ERROR, "ERR 4C16: " + "redirection requested but disabled\n"); + invalid = TRUE; + } + + if (cpi->redir_lid == 0) { + /* GID redirection: get PathRecord information */ + OSM_LOG(pm->log, OSM_LOG_ERROR, "ERR 4C21: " + "GID redirection not currently supported\n"); goto Exit; } @@ -1144,14 +1302,23 @@ static void osm_pc_rcv_process(void *context, void *data) p_mon_node->redir_tbl_size); goto Exit; } + p_mon_node->redir_port[port].redirection = TRUE; + p_mon_node->redir_port[port].invalid = invalid; + memcpy(&p_mon_node->redir_port[port].redir_gid, + &cpi->redir_gid, sizeof(ib_gid_t)); p_mon_node->redir_port[port].redir_lid = cpi->redir_lid; p_mon_node->redir_port[port].redir_qp = cpi->redir_qp; + p_mon_node->redir_port[port].redir_pkey = cpi->redir_pkey; + p_mon_node->redir_port[port].redir_pkey_ix = pkey_ix; cl_plock_release(pm->lock); + if (invalid) + goto Exit; + /* Finally, reissue the query to the redirected location */ status = osm_perfmgr_send_pc_mad(pm, cpi->redir_lid, cpi->redir_qp, - port, + pkey_ix, port, mad_context->perfmgr_context. mad_method, mad_context); if (status != IB_SUCCESS) @@ -1234,6 +1401,7 @@ osm_perfmgr_init(osm_perfmgr_t * const pm, osm_opensm_t *osm, pm->sweep_time_s = p_opt->perfmgr_sweep_time_s; pm->max_outstanding_queries = p_opt->perfmgr_max_outstanding_queries; pm->osm = osm; + pm->local_port = -1; status = cl_timer_init(&pm->sweep_timer, perfmgr_sweep, pm); if (status != IB_SUCCESS) From hnrose at comcast.net Wed Mar 11 06:29:19 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 11 Mar 2009 08:29:19 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] infiniband-diags/vendstat.c: Add missing mad_rpc_close_port call Message-ID: <20090311132919.GA23957@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/infiniband-diags/src/vendstat.c b/infiniband-diags/src/vendstat.c index d001a01..0f09039 100644 --- a/infiniband-diags/src/vendstat.c +++ b/infiniband-diags/src/vendstat.c @@ -247,5 +247,6 @@ int main(int argc, char **argv) ntohl(cs->record[i].data)); } + mad_rpc_close_port(srcport); exit(0); } From hnrose at comcast.net Wed Mar 11 06:53:20 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 11 Mar 2009 08:53:20 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] infiniband-diags/mcm_rereg_test.c: Add missing mad_rpc_close_port call Message-ID: <20090311135320.GA29870@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/infiniband-diags/src/mcm_rereg_test.c b/infiniband-diags/src/mcm_rereg_test.c index de5ea90..d087e4a 100644 --- a/infiniband-diags/src/mcm_rereg_test.c +++ b/infiniband-diags/src/mcm_rereg_test.c @@ -492,9 +492,13 @@ int main(int argc, char **argv) mad = umad_get_mad(umad); free(umad); +#if 1 + mad_rpc_close_port(srcport); +#else umad_unregister(port, agent); umad_close_port(port); umad_done(); +#endif return 0; } From ogerlitz at voltaire.com Wed Mar 11 06:36:17 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 11 Mar 2009 15:36:17 +0200 (IST) Subject: [ofa-general] merging madeye into mainline Message-ID: Hi Sean, Have you ever considered to push the madeye module (below) into the kernel to ease with fabric debugging? I have tested it now against Linus tree and it works fine. Or. Index: linus-linux-2.6/drivers/infiniband/Kconfig =================================================================== --- linus-linux-2.6.orig/drivers/infiniband/Kconfig +++ linus-linux-2.6/drivers/infiniband/Kconfig @@ -40,6 +40,12 @@ config INFINIBAND_ADDR_TRANS depends on !(INFINIBAND = y && IPV6 = m) default y +config INFINIBAND_MADEYE + tristate "InfiniBand MAD viewer support" + depends on INFINIBAND + ---help--- + viewer for InfiniBand Management Datagrams (MAD) + source "drivers/infiniband/hw/mthca/Kconfig" source "drivers/infiniband/hw/ipath/Kconfig" source "drivers/infiniband/hw/ehca/Kconfig" Index: linus-linux-2.6/drivers/infiniband/Makefile =================================================================== --- linus-linux-2.6.orig/drivers/infiniband/Makefile +++ linus-linux-2.6/drivers/infiniband/Makefile @@ -9,3 +9,4 @@ obj-$(CONFIG_INFINIBAND_NES) += hw/nes/ obj-$(CONFIG_INFINIBAND_IPOIB) += ulp/ipoib/ obj-$(CONFIG_INFINIBAND_SRP) += ulp/srp/ obj-$(CONFIG_INFINIBAND_ISER) += ulp/iser/ +obj-$(CONFIG_INFINIBAND_MADEYE) += util/ Index: linus-linux-2.6/drivers/infiniband/util/Makefile =================================================================== --- /dev/null +++ linus-linux-2.6/drivers/infiniband/util/Makefile @@ -0,0 +1,3 @@ +obj-$(CONFIG_INFINIBAND_MADEYE) += ib_madeye.o + +ib_madeye-y := madeye.o Index: linus-linux-2.6/drivers/infiniband/util/madeye.c =================================================================== --- /dev/null +++ linus-linux-2.6/drivers/infiniband/util/madeye.c @@ -0,0 +1,593 @@ +/* + * Copyright (c) 2004, 2005 Intel Corporation. All rights reserved. + * Copyright (c) 2005, 2006 Voltaire Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directorY of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + * $Id$ + */ +#include +#include +#include + +#include +#include +#include + +MODULE_AUTHOR("Sean Hefty"); +MODULE_DESCRIPTION("InfiniBand MAD viewer"); +MODULE_LICENSE("Dual BSD/GPL"); + +static void madeye_remove_one(struct ib_device *device); +static void madeye_add_one(struct ib_device *device); + +static struct ib_client madeye_client = { + .name = "madeye", + .add = madeye_add_one, + .remove = madeye_remove_one +}; + +struct madeye_port { + struct ib_mad_agent *smi_agent; + struct ib_mad_agent *gsi_agent; +}; + +static int smp = 1; +static int gmp = 1; +static int mgmt_class = 0; +static int attr_id = 0; +static int data = 0; + +module_param(smp, int, 0444); +module_param(gmp, int, 0444); +module_param(mgmt_class, int, 0444); +module_param(attr_id, int, 0444); +module_param(data, int, 0444); + +MODULE_PARM_DESC(smp, "Display all SMPs (default=1)"); +MODULE_PARM_DESC(gmp, "Display all GMPs (default=1)"); +MODULE_PARM_DESC(mgmt_class, "Display all MADs of specified class (default=0)"); +MODULE_PARM_DESC(attr_id, "Display add MADs of specified attribute ID (default=0)"); +MODULE_PARM_DESC(data, "Display data area of MADs (default=0)"); + +static char * get_class_name(u8 mgmt_class) +{ + switch(mgmt_class) { + case IB_MGMT_CLASS_SUBN_LID_ROUTED: + return "LID routed SMP"; + case IB_MGMT_CLASS_SUBN_DIRECTED_ROUTE: + return "Directed route SMP"; + case IB_MGMT_CLASS_SUBN_ADM: + return "Subnet admin."; + case IB_MGMT_CLASS_PERF_MGMT: + return "Perf. mgmt."; + case IB_MGMT_CLASS_BM: + return "Baseboard mgmt."; + case IB_MGMT_CLASS_DEVICE_MGMT: + return "Device mgmt."; + case IB_MGMT_CLASS_CM: + return "Comm. mgmt."; + case IB_MGMT_CLASS_SNMP: + return "SNMP"; + default: + return "Unknown vendor/application"; + } +} + +static char * get_method_name(u8 mgmt_class, u8 method) +{ + switch(method) { + case IB_MGMT_METHOD_GET: + return "Get"; + case IB_MGMT_METHOD_SET: + return "Set"; + case IB_MGMT_METHOD_GET_RESP: + return "Get response"; + case IB_MGMT_METHOD_SEND: + return "Send"; + case IB_MGMT_METHOD_SEND | IB_MGMT_METHOD_RESP: + return "Send response"; + case IB_MGMT_METHOD_TRAP: + return "Trap"; + case IB_MGMT_METHOD_REPORT: + return "Report"; + case IB_MGMT_METHOD_REPORT_RESP: + return "Report response"; + case IB_MGMT_METHOD_TRAP_REPRESS: + return "Trap repress"; + default: + break; + } + + switch (mgmt_class) { + case IB_MGMT_CLASS_SUBN_ADM: + switch (method) { + case IB_SA_METHOD_GET_TABLE: + return "Get table"; + case IB_SA_METHOD_GET_TABLE_RESP: + return "Get table response"; + case IB_SA_METHOD_DELETE: + return "Delete"; + case IB_SA_METHOD_DELETE_RESP: + return "Delete response"; + case IB_SA_METHOD_GET_MULTI: + return "Get Multi"; + case IB_SA_METHOD_GET_MULTI_RESP: + return "Get Multi response"; + case IB_SA_METHOD_GET_TRACE_TBL: + return "Get Trace Table response"; + default: + break; + } + default: + break; + } + + return "Unknown"; +} + +static void print_status_details(u16 status) +{ + if (status & 0x0001) + printk(" busy\n"); + if (status & 0x0002) + printk(" redirection required\n"); + switch((status & 0x001C) >> 2) { + case 1: + printk(" bad version\n"); + break; + case 2: + printk(" method not supported\n"); + break; + case 3: + printk(" method/attribute combo not supported\n"); + break; + case 7: + printk(" invalid attribute/modifier value\n"); + break; + } +} + +static char * get_sa_attr(__be16 attr) +{ + switch(attr) { + case IB_SA_ATTR_CLASS_PORTINFO: + return "Class Port Info"; + case IB_SA_ATTR_NOTICE: + return "Notice"; + case IB_SA_ATTR_INFORM_INFO: + return "Inform Info"; + case IB_SA_ATTR_NODE_REC: + return "Node Record"; + case IB_SA_ATTR_PORT_INFO_REC: + return "PortInfo Record"; + case IB_SA_ATTR_SL2VL_REC: + return "SL to VL Record"; + case IB_SA_ATTR_SWITCH_REC: + return "Switch Record"; + case IB_SA_ATTR_LINEAR_FDB_REC: + return "Linear FDB Record"; + case IB_SA_ATTR_RANDOM_FDB_REC: + return "Random FDB Record"; + case IB_SA_ATTR_MCAST_FDB_REC: + return "Multicast FDB Record"; + case IB_SA_ATTR_SM_INFO_REC: + return "SM Info Record"; + case IB_SA_ATTR_LINK_REC: + return "Link Record"; + case IB_SA_ATTR_GUID_INFO_REC: + return "Guid Info Record"; + case IB_SA_ATTR_SERVICE_REC: + return "Service Record"; + case IB_SA_ATTR_PARTITION_REC: + return "Partition Record"; + case IB_SA_ATTR_PATH_REC: + return "Path Record"; + case IB_SA_ATTR_VL_ARB_REC: + return "VL Arb Record"; + case IB_SA_ATTR_MC_MEMBER_REC: + return "MC Member Record"; + case IB_SA_ATTR_TRACE_REC: + return "Trace Record"; + case IB_SA_ATTR_MULTI_PATH_REC: + return "Multi Path Record"; + case IB_SA_ATTR_SERVICE_ASSOC_REC: + return "Service Assoc Record"; + case IB_SA_ATTR_INFORM_INFO_REC: + return "Inform Info Record"; + default: + return ""; + } +} + +static void print_mad_hdr(struct ib_mad_hdr *mad_hdr) +{ + printk("MAD version....0x%01x\n", mad_hdr->base_version); + printk("Class..........0x%01x (%s)\n", mad_hdr->mgmt_class, + get_class_name(mad_hdr->mgmt_class)); + printk("Class version..0x%01x\n", mad_hdr->class_version); + printk("Method.........0x%01x (%s)\n", mad_hdr->method, + get_method_name(mad_hdr->mgmt_class, mad_hdr->method)); + printk("Status.........0x%02x\n", be16_to_cpu(mad_hdr->status)); + if (mad_hdr->status) + print_status_details(be16_to_cpu(mad_hdr->status)); + printk("Class specific.0x%02x\n", be16_to_cpu(mad_hdr->class_specific)); + printk("Trans ID.......0x%llx\n", + (unsigned long long)be64_to_cpu(mad_hdr->tid)); + if (mad_hdr->mgmt_class == IB_MGMT_CLASS_SUBN_ADM) + printk("Attr ID........0x%02x (%s)\n", + be16_to_cpu(mad_hdr->attr_id), + get_sa_attr(be16_to_cpu(mad_hdr->attr_id))); + else + printk("Attr ID........0x%02x\n", + be16_to_cpu(mad_hdr->attr_id)); + printk("Attr modifier..0x%04x\n", be32_to_cpu(mad_hdr->attr_mod)); +} + +static char * get_rmpp_type(u8 rmpp_type) +{ + switch (rmpp_type) { + case IB_MGMT_RMPP_TYPE_DATA: + return "Data"; + case IB_MGMT_RMPP_TYPE_ACK: + return "Ack"; + case IB_MGMT_RMPP_TYPE_STOP: + return "Stop"; + case IB_MGMT_RMPP_TYPE_ABORT: + return "Abort"; + default: + return "Unknown"; + } +} + +static char * get_rmpp_flags(u8 rmpp_flags) +{ + if (rmpp_flags & IB_MGMT_RMPP_FLAG_ACTIVE) + if (rmpp_flags & IB_MGMT_RMPP_FLAG_FIRST) + if (rmpp_flags & IB_MGMT_RMPP_FLAG_LAST) + return "Active - First & Last"; + else + return "Active - First"; + else + if (rmpp_flags & IB_MGMT_RMPP_FLAG_LAST) + return "Active - Last"; + else + return "Active"; + else + return "Inactive"; +} + +static void print_rmpp_hdr(struct ib_rmpp_hdr *rmpp_hdr) +{ + printk("RMPP version...0x%01x\n", rmpp_hdr->rmpp_version); + printk("RMPP type......0x%01x (%s)\n", rmpp_hdr->rmpp_type, + get_rmpp_type(rmpp_hdr->rmpp_type)); + printk("RMPP RRespTime.0x%01x\n", ib_get_rmpp_resptime(rmpp_hdr)); + printk("RMPP flags.....0x%01x (%s)\n", ib_get_rmpp_flags(rmpp_hdr), + get_rmpp_flags(ib_get_rmpp_flags(rmpp_hdr))); + printk("RMPP status....0x%01x\n", rmpp_hdr->rmpp_status); + printk("Seg number.....0x%04x\n", be32_to_cpu(rmpp_hdr->seg_num)); + switch (rmpp_hdr->rmpp_type) { + case IB_MGMT_RMPP_TYPE_DATA: + printk("Payload len....0x%04x\n", + be32_to_cpu(rmpp_hdr->paylen_newwin)); + break; + case IB_MGMT_RMPP_TYPE_ACK: + printk("New window.....0x%04x\n", + be32_to_cpu(rmpp_hdr->paylen_newwin)); + break; + default: + printk("Data 2.........0x%04x\n", + be32_to_cpu(rmpp_hdr->paylen_newwin)); + break; + } +} + +static char * get_smp_attr(__be16 attr) +{ + switch (attr) { + case IB_SMP_ATTR_NOTICE: + return "notice"; + case IB_SMP_ATTR_NODE_DESC: + return "node description"; + case IB_SMP_ATTR_NODE_INFO: + return "node info"; + case IB_SMP_ATTR_SWITCH_INFO: + return "switch info"; + case IB_SMP_ATTR_GUID_INFO: + return "GUID info"; + case IB_SMP_ATTR_PORT_INFO: + return "port info"; + case IB_SMP_ATTR_PKEY_TABLE: + return "pkey table"; + case IB_SMP_ATTR_SL_TO_VL_TABLE: + return "SL to VL table"; + case IB_SMP_ATTR_VL_ARB_TABLE: + return "VL arbitration table"; + case IB_SMP_ATTR_LINEAR_FORWARD_TABLE: + return "linear forwarding table"; + case IB_SMP_ATTR_RANDOM_FORWARD_TABLE: + return "random forward table"; + case IB_SMP_ATTR_MCAST_FORWARD_TABLE: + return "multicast forward table"; + case IB_SMP_ATTR_SM_INFO: + return "SM info"; + case IB_SMP_ATTR_VENDOR_DIAG: + return "vendor diags"; + case IB_SMP_ATTR_LED_INFO: + return "LED info"; + default: + return ""; + } +} + +static void print_smp(struct ib_smp *smp) +{ + int i; + + printk("MAD version....0x%01x\n", smp->base_version); + printk("Class..........0x%01x (%s)\n", smp->mgmt_class, + get_class_name(smp->mgmt_class)); + printk("Class version..0x%01x\n", smp->class_version); + printk("Method.........0x%01x (%s)\n", smp->method, + get_method_name(smp->mgmt_class, smp->method)); + printk("Status.........0x%02x\n", be16_to_cpu(smp->status)); + if (smp->status) + print_status_details(be16_to_cpu(smp->status)); + printk("Hop pointer....0x%01x\n", smp->hop_ptr); + printk("Hop counter....0x%01x\n", smp->hop_cnt); + printk("Trans ID.......0x%llx\n", + (unsigned long long)be64_to_cpu(smp->tid)); + printk("Attr ID........0x%02x (%s)\n", be16_to_cpu(smp->attr_id), + get_smp_attr(smp->attr_id)); + printk("Attr modifier..0x%04x\n", be32_to_cpu(smp->attr_mod)); + + printk("Mkey...........0x%llx\n", + (unsigned long long)be64_to_cpu(smp->mkey)); + printk("DR SLID........0x%02x\n", be16_to_cpu(smp->dr_slid)); + printk("DR DLID........0x%02x", be16_to_cpu(smp->dr_dlid)); + + if (data) { + for (i = 0; i < IB_SMP_DATA_SIZE; i++) { + if (i % 16 == 0) + printk("\nSMP Data......."); + printk("%01x ", smp->data[i]); + } + for (i = 0; i < IB_SMP_MAX_PATH_HOPS; i++) { + if (i % 16 == 0) + printk("\nInitial path..."); + printk("%01x ", smp->initial_path[i]); + } + for (i = 0; i < IB_SMP_MAX_PATH_HOPS; i++) { + if (i % 16 == 0) + printk("\nReturn path...."); + printk("%01x ", smp->return_path[i]); + } + } + printk("\n"); +} + +static void snoop_smi_handler(struct ib_mad_agent *mad_agent, + struct ib_mad_send_buf *send_buf, + struct ib_mad_send_wc *mad_send_wc) +{ + struct ib_mad_hdr *hdr = send_buf->mad; + + if (!smp && hdr->mgmt_class != mgmt_class) + return; + if (attr_id && hdr->attr_id != attr_id) + return; + + printk("Madeye:sent SMP\n"); + print_smp(send_buf->mad); +} + +static void recv_smi_handler(struct ib_mad_agent *mad_agent, + struct ib_mad_recv_wc *mad_recv_wc) +{ + if (!smp && mad_recv_wc->recv_buf.mad->mad_hdr.mgmt_class != mgmt_class) + return; + if (attr_id && mad_recv_wc->recv_buf.mad->mad_hdr.attr_id != attr_id) + return; + + printk("Madeye:recv SMP\n"); + print_smp((struct ib_smp *)&mad_recv_wc->recv_buf.mad->mad_hdr); +} + +static int is_rmpp_mad(struct ib_mad_hdr *mad_hdr) +{ + if (mad_hdr->mgmt_class == IB_MGMT_CLASS_SUBN_ADM) { + switch (mad_hdr->method) { + case IB_SA_METHOD_GET_TABLE: + case IB_SA_METHOD_GET_TABLE_RESP: + case IB_SA_METHOD_GET_MULTI_RESP: + return 1; + default: + break; + } + } else if ((mad_hdr->mgmt_class >= IB_MGMT_CLASS_VENDOR_RANGE2_START) && + (mad_hdr->mgmt_class <= IB_MGMT_CLASS_VENDOR_RANGE2_END)) + return 1; + + return 0; +} + +static void snoop_gsi_handler(struct ib_mad_agent *mad_agent, + struct ib_mad_send_buf *send_buf, + struct ib_mad_send_wc *mad_send_wc) +{ + struct ib_mad_hdr *hdr = send_buf->mad; + + if (!gmp && hdr->mgmt_class != mgmt_class) + return; + if (attr_id && hdr->attr_id != attr_id) + return; + + printk("Madeye:sent GMP\n"); + print_mad_hdr(hdr); + + if (is_rmpp_mad(hdr)) + print_rmpp_hdr(&((struct ib_rmpp_mad *) hdr)->rmpp_hdr); +} + +static void recv_gsi_handler(struct ib_mad_agent *mad_agent, + struct ib_mad_recv_wc *mad_recv_wc) +{ + struct ib_mad_hdr *hdr = &mad_recv_wc->recv_buf.mad->mad_hdr; + struct ib_rmpp_mad *mad = NULL; + struct ib_sa_mad *sa_mad; + struct ib_vendor_mad *vendor_mad; + u8 *mad_data; + int i, j; + + if (!gmp && hdr->mgmt_class != mgmt_class) + return; + if (attr_id && mad_recv_wc->recv_buf.mad->mad_hdr.attr_id != attr_id) + return; + + printk("Madeye:recv GMP\n"); + print_mad_hdr(hdr); + + if (is_rmpp_mad(hdr)) { + mad = (struct ib_rmpp_mad *) hdr; + print_rmpp_hdr(&mad->rmpp_hdr); + } + + if (data) { + if (hdr->mgmt_class == IB_MGMT_CLASS_SUBN_ADM) { + j = IB_MGMT_SA_DATA; + /* Display SA header */ + if (is_rmpp_mad(hdr) && + mad->rmpp_hdr.rmpp_type != IB_MGMT_RMPP_TYPE_DATA) + return; + sa_mad = (struct ib_sa_mad *) + &mad_recv_wc->recv_buf.mad; + mad_data = sa_mad->data; + } else { + if (is_rmpp_mad(hdr)) { + j = IB_MGMT_VENDOR_DATA; + /* Display OUI */ + vendor_mad = (struct ib_vendor_mad *) + &mad_recv_wc->recv_buf.mad; + printk("Vendor OUI......%01x %01x %01x\n", + vendor_mad->oui[0], + vendor_mad->oui[1], + vendor_mad->oui[2]); + mad_data = vendor_mad->data; + } else { + j = IB_MGMT_MAD_DATA; + mad_data = mad_recv_wc->recv_buf.mad->data; + } + } + for (i = 0; i < j; i++) { + if (i % 16 == 0) + printk("\nData..........."); + printk("%01x ", mad_data[i]); + } + printk("\n"); + } +} + +static void madeye_add_one(struct ib_device *device) +{ + struct madeye_port *port; + int reg_flags; + u8 i, s, e; + + if (device->node_type == RDMA_NODE_IB_SWITCH) { + s = 0; + e = 0; + } else { + s = 1; + e = device->phys_port_cnt; + } + + port = kmalloc(sizeof *port * (e - s + 1), GFP_KERNEL); + if (!port) + goto out; + + reg_flags = IB_MAD_SNOOP_SEND_COMPLETIONS | IB_MAD_SNOOP_RECVS; + for (i = 0; i <= e - s; i++) { + port[i].smi_agent = ib_register_mad_snoop(device, i + s, + IB_QPT_SMI, + reg_flags, + snoop_smi_handler, + recv_smi_handler, + &port[i]); + port[i].gsi_agent = ib_register_mad_snoop(device, i + s, + IB_QPT_GSI, + reg_flags, + snoop_gsi_handler, + recv_gsi_handler, + &port[i]); + } + +out: + ib_set_client_data(device, &madeye_client, port); +} + +static void madeye_remove_one(struct ib_device *device) +{ + struct madeye_port *port; + int i, s, e; + + port = (struct madeye_port *) + ib_get_client_data(device, &madeye_client); + if (!port) + return; + + if (device->node_type == RDMA_NODE_IB_SWITCH) { + s = 0; + e = 0; + } else { + s = 1; + e = device->phys_port_cnt; + } + + for (i = 0; i <= e - s; i++) { + if (!IS_ERR(port[i].smi_agent)) + ib_unregister_mad_agent(port[i].smi_agent); + if (!IS_ERR(port[i].gsi_agent)) + ib_unregister_mad_agent(port[i].gsi_agent); + } + kfree(port); +} + +static int __init ib_madeye_init(void) +{ + return ib_register_client(&madeye_client); +} + +static void __exit ib_madeye_cleanup(void) +{ + ib_unregister_client(&madeye_client); +} + +module_init(ib_madeye_init); +module_exit(ib_madeye_cleanup); From ben at lanl.gov Wed Mar 11 07:35:49 2009 From: ben at lanl.gov (Ben) Date: Wed, 11 Mar 2009 08:35:49 -0600 Subject: [ofa-general] ipoib performanec at smaller mtu In-Reply-To: <1E3DCD1C63492545881FACB6063A57C1038A83EC@mtiexch01> References: <59748.128.165.248.31.1236640753.squirrel@webmail.lanl.gov> <1E3DCD1C63492545881FACB6063A57C1038A80F5@mtiexch01> <03EC693D-38BE-4947-8D7A-7C8CC3255815@lanl.gov> <1E3DCD1C63492545881FACB6063A57C1038A83EC@mtiexch01> Message-ID: <28F92B3C-8E1E-4B3A-9A41-0ABD4B436376@lanl.gov> here is the iperf test: [ ID] Interval Transfer Bandwidth [ 9] 0.0-10.0 sec 244 MBytes 205 Mbits/sec [ ID] Interval Transfer Bandwidth [ 5] 0.0-10.0 sec 222 MBytes 187 Mbits/sec [ ID] Interval Transfer Bandwidth [ 10] 0.0-10.0 sec 190 MBytes 160 Mbits/sec [ ID] Interval Transfer Bandwidth [ 6] 0.0-10.0 sec 406 MBytes 340 Mbits/sec [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 204 MBytes 171 Mbits/sec [ ID] Interval Transfer Bandwidth [ 8] 0.0-10.0 sec 229 MBytes 191 Mbits/sec [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 222 MBytes 186 Mbits/sec [ ID] Interval Transfer Bandwidth [ 7] 0.0-10.6 sec 147 MBytes 116 Mbits/sec [SUM] 0.0-10.6 sec 1.82 GBytes 1.47 Gbits/sec On a RHEL5 with OFED 1.3 machine with similar ib cards, I can get about 3x this. -Ben On Mar 10, 2009, at 6:20 PM, Boris Shpolyansky wrote: > Ben, > > What is the 8-stream BW you are seeing with IPoIB when using datagram > mode and MTU=1500? > > Boris Shpolyansky > Sr. Member of Technical Staff, Applications > > Mellanox Technologies Inc. > 350 Oakmead Parkway, Suite 100 > Sunnyvale, CA 94085 > Tel.: (408) 916 0014 > Fax: (408) 585 0314 > Cell: (408) 834 9365 > www.mellanox.com > > -----Original Message----- > From: Ben [mailto:ben at lanl.gov] > Sent: Monday, March 09, 2009 5:19 PM > To: Boris Shpolyansky > Cc: general at lists.openfabrics.org > Subject: Re: [ofa-general] ipoib performanec at smaller mtu > > I have a similar setup in datagram, and don't see a big difference. > Are these numbers expected? I guess I would like to understand the > limitations. It seams like these numbers are a bit below what the > hardware is capable of even with lro off. > > -Ben > > On Mar 9, 2009, at 6:09 PM, Boris Shpolyansky wrote: > >> I think for small MTU sizes you will see better performance with >> IPoIB >> in datagram mode. >> >> Boris Shpolyansky >> Sr. Member of Technical Staff, Applications >> >> Mellanox Technologies Inc. >> 350 Oakmead Parkway, Suite 100 >> Sunnyvale, CA 94085 >> Tel.: (408) 916 0014 >> Fax: (408) 585 0314 >> Cell: (408) 834 9365 >> www.mellanox.com >> >> -----Original Message----- >> From: general-bounces at lists.openfabrics.org >> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of >> Benjamin M. >> McClelland >> Sent: Monday, March 09, 2009 4:19 PM >> To: general at lists.openfabrics.org >> Subject: [ofa-general] ipoib performanec at smaller mtu >> >> I was curious what other people are seeing for 1500 mtu >> performance on >> ipoib. Current tests are showing that I get about 150-175MB/s no >> matter >> how many streams. If I go up to higher mtu, then performance >> significantly increases. Due to the storage network that this ib is >> routing to, we are currently limited to 1500 mtu. >> >> ofed 1.4.1, connected mode, dual opteron dual core 1.8GHz, Mellanox >> Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0) fw >> 2.6.000, voltaire sm on switch >> >> lro disabled, send_queue_size=256 >> >> thanks, >> -Ben >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general > From swise at opengridcomputing.com Wed Mar 11 08:07:12 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 11 Mar 2009 10:07:12 -0500 Subject: [ofa-general] tcp/rdma port unification patch in librdmacm In-Reply-To: <49B76CD5.705@voltaire.com> References: <49B17E15.4050406@opengridcomputing.com> <49B76CD5.705@voltaire.com> Message-ID: <49B7D3A0.5060307@opengridcomputing.com> Or Gerlitz wrote: > Steve Wise wrote: >> It seems like kernel RDMA protocols are all getting IANA-assigned >> port numbers that alleviates the need for TCP/RDMA port unification. >> NFSRDMA and RDS are two. But user applications like mvapich2 and >> other mpis often let the stack choose a local port number on-which to >> bind/listen. > Hi Steve, > > You have mentioned the listeners, but what about the ports used for > the connection itself? isn't it correct that when the TOE stack > accepts the rNFS connection request to port X - still a different port > Y would be used for the connection itself ? but if the Linux TCP stack > would have (say) iperf bind to port Y then you are again in troubles - > am I missing anything here? > > Or. > There is no "different port Y". I don't understand what you mean. From swise at opengridcomputing.com Wed Mar 11 08:09:16 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 11 Mar 2009 10:09:16 -0500 Subject: [ofa-general] RE: tcp/rdma port unification patch in librdmacm In-Reply-To: <49B767DB.2050700@voltaire.com> References: <49B17E15.4050406@opengridcomputing.com> <811553D6D34A4E6F9AB686CA2E441D35@amr.corp.intel.com> <49B61C5D.6060900@Voltaire.com> <3281521B4F4A415A846A9F2E8AF9C21F@amr.corp.intel.com> <49B767DB.2050700@voltaire.com> Message-ID: <49B7D41C.4030707@opengridcomputing.com> Or Gerlitz wrote: > Sean Hefty wrote: >> I thought about that, but didn't know whether it was in use. I agree >> that >> existing apps shouldn't break. (I was thinking more along the lines >> of adding a >> new call that would make this behavior explicit, but haven't taken >> the time to >> really study the details.) >> > > Yes, lets not break existing apps such as stgt. I'm fine with adding > new call. > Or maybe map PS_TCP port space to the be the unified space, and have a new PS_RDMA or PS_IB port space that doesn't? >> Does anyone know if a kernel patch to fix this has been accepted >> directly into >> the distros? >> > > Sorry, but I wasn't sure to follow what you mean by "fix this" ... did > you refer to kernel apps that don't use different port numbers for > their TCP vs RDMA listeners, or you referred to the rdma_cm patch > which is not merged into mainline? > I haven't pushed to get a distro to pull in the cma patch. I assumed since it wasn't destined for upstream, that they wouldn't accept it. >> Also, does anything keep MPI from doing exactly what we're discussing >> in the >> application as part of using the librdmacm? (Besides having all apps >> duplicate this >> > no, nothing prevents an app to open/bound a socket as long as the port > is available and its ulimit allow to open the number of sockets it > wants. This actually somehow brings me back to square one with regard > to the actual problem, so I'll ask you guys a question as a reply to > earlier post on this thread... > Nothing prevents this, but why force OpenMPI, MVAPICH2, HP MPI, Intel MPI, and Scali MPI to all do this? Steve. From or.gerlitz at gmail.com Wed Mar 11 08:19:23 2009 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Wed, 11 Mar 2009 17:19:23 +0200 Subject: ***SPAM*** Re: [ofa-general] tcp/rdma port unification patch in librdmacm In-Reply-To: <49B7D3A0.5060307@opengridcomputing.com> References: <49B17E15.4050406@opengridcomputing.com> <49B76CD5.705@voltaire.com> <49B7D3A0.5060307@opengridcomputing.com> Message-ID: <15ddcffd0903110819s3b83cec0vb87ee6114496858e@mail.gmail.com> > > You have mentioned the listeners, but what about the ports used for the >> connection itself? >> > > There is no "different port Y". I don't understand what you mean. mmm, think for example on the active side (rNFS client) say the TOE stack assigned it port Y and also the Linux TCP stack assigned port Y to the netperf client then when the server node sends data to the client IP port Y - who is going to get it? Or. -------------- next part -------------- An HTML attachment was scrubbed... URL: From swise at opengridcomputing.com Wed Mar 11 08:28:25 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 11 Mar 2009 10:28:25 -0500 Subject: [ofa-general] tcp/rdma port unification patch in librdmacm In-Reply-To: <15ddcffd0903110819s3b83cec0vb87ee6114496858e@mail.gmail.com> References: <49B17E15.4050406@opengridcomputing.com> <49B76CD5.705@voltaire.com> <49B7D3A0.5060307@opengridcomputing.com> <15ddcffd0903110819s3b83cec0vb87ee6114496858e@mail.gmail.com> Message-ID: <49B7D899.1030601@opengridcomputing.com> Or Gerlitz wrote: > > You have mentioned the listeners, but what about the ports > used for the connection itself? > > > There is no "different port Y". I don't understand what you mean. > > > > mmm, think for example on the active side (rNFS client) say the TOE > stack assigned it port Y and also the Linux TCP stack assigned port Y > to the netperf client then when the server node sends data to the > client IP port Y - who is going to get it? > > Or. Depends on the 4-tuple. The demux done in the TOE hw uses src addr/port and dst addr/port. So since the rNFS connection will always have the reserved rNFS port for the server side, the 4-tuple will always be unique. The netperf traffic would go up the NIC side and the rNFS traffic up the TOE side... Steve. From weiny2 at llnl.gov Wed Mar 11 09:03:37 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 11 Mar 2009 09:03:37 -0700 Subject: [ofa-general] ***SPAM*** [PATCH] infiniband-diags/mcm_rereg_test.c: Add missing mad_rpc_close_port call In-Reply-To: <20090311135320.GA29870@comcast.net> References: <20090311135320.GA29870@comcast.net> Message-ID: <20090311090337.ec293807.weiny2@llnl.gov> On Wed, 11 Mar 2009 08:53:20 -0500 Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock > > --- > diff --git a/infiniband-diags/src/mcm_rereg_test.c b/infiniband-diags/src/mcm_rereg_test.c > index de5ea90..d087e4a 100644 > --- a/infiniband-diags/src/mcm_rereg_test.c > +++ b/infiniband-diags/src/mcm_rereg_test.c > @@ -492,9 +492,13 @@ int main(int argc, char **argv) > mad = umad_get_mad(umad); > > free(umad); > +#if 1 > + mad_rpc_close_port(srcport); > +#else > umad_unregister(port, agent); > umad_close_port(port); > umad_done(); > +#endif Why use "#if 1" here? Why not just replace the umad calls with mad_rpc_close_port? Ira > > return 0; > } > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http:// lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http:// openib.org/mailman/listinfo/openib-general > -- Ira Weiny Math Programer/Computer Scientist Larence Livermore National Lab weiny2 at llnl.gov From weiny2 at llnl.gov Wed Mar 11 09:09:41 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 11 Mar 2009 09:09:41 -0700 Subject: [ofa-general] [PATCH 11/10] libibmad:infiniband-diags: deprecate madrpc_set_[retries|timeout] WAS: [PATCH 1/10] libibmad: Clean up "new" interface In-Reply-To: <20090311102941.GB22085@sashak.voltaire.com> References: <20090219190525.322681b8.weiny2@llnl.gov> <20090220143402.c3b23b0a.weiny2@llnl.gov> <20090301072622.GG3936@sashak.voltaire.com> <20090302111101.bc822db8.weiny2@llnl.gov> <20090307191647.GJ9673@sashak.voltaire.com> <20090310163426.63d67ff2.weiny2@llnl.gov> <20090311102941.GB22085@sashak.voltaire.com> Message-ID: <20090311090941.fde8d90c.weiny2@llnl.gov> On Wed, 11 Mar 2009 12:29:41 +0200 Sasha Khapyorsky wrote: > On 16:34 Tue 10 Mar , Ira Weiny wrote: > > > > I will look at documenting the above as technically it is possible to change > > the timeout on various ports by setting the parameters appropriately on each > > call. I don't like this going forward but I guess it is doable without > > changing the interface. > > And what do you say about the patch below? It doesn't change the current > diags behavior and provides per port timeout/retries setup. The global > parameters and rpc's fields remain to be same, and we don;t need to > deprecate madrpc_set_timeout/retires(). OK! I miss-understood you. I thought you did not want to add another set of timeout/retry values. The below is just fine with me. Do you ever see getting rid of the global default? Ira > > Sasha > > > >From 25060f5a7243a1bae6a7e7e9b2b79051ab0473d9 Mon Sep 17 00:00:00 2001 > From: Sasha Khapyorsky > Date: Wed, 11 Mar 2009 12:15:54 +0200 > Subject: [PATCH] libibmad: per port timeout and retires setup > > This adds functions mad_rpc_set_timeout() and mad_rpc_set_retries() > which provides possibility to specify timeout and retries for mad rpc > processing per port. This doesn't change the default behavior. > > Signed-off-by: Sasha Khapyorsky > --- > libibmad/include/infiniband/mad.h | 2 ++ > libibmad/src/libibmad.map | 2 ++ > libibmad/src/mad_internal.h | 1 + > libibmad/src/rpc.c | 37 +++++++++++++++++++++++++++---------- > 4 files changed, 32 insertions(+), 10 deletions(-) > > diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h > index 064cbb7..9b398fa 100644 > --- a/libibmad/include/infiniband/mad.h > +++ b/libibmad/include/infiniband/mad.h > @@ -742,6 +742,8 @@ MAD_EXPORT void *mad_rpc(const struct ibmad_port *srcport, ib_rpc_t * rpc, ib_po > MAD_EXPORT void *mad_rpc_rmpp(const struct ibmad_port *srcport, ib_rpc_t * rpc, ib_portid_t * dport, > ib_rmpp_hdr_t * rmpp, void *data); > MAD_EXPORT int mad_rpc_portid(struct ibmad_port *srcport); > +MAD_EXPORT void mad_rpc_set_retries(struct ibmad_port *port, int retries); > +MAD_EXPORT void mad_rpc_set_timeout(struct ibmad_port *port, int timeout); > > /* register.c */ > MAD_EXPORT int mad_register_port_client(int port_id, int mgmt, > diff --git a/libibmad/src/libibmad.map b/libibmad/src/libibmad.map > index 0be7a92..10c7cd0 100644 > --- a/libibmad/src/libibmad.map > +++ b/libibmad/src/libibmad.map > @@ -68,6 +68,8 @@ IBMAD_1.3 { > mad_rpc; > mad_rpc_rmpp; > mad_rpc_portid; > + mad_rpc_set_retries; > + mad_rpc_set_timeout; > madrpc; > madrpc_def_timeout; > madrpc_init; > diff --git a/libibmad/src/mad_internal.h b/libibmad/src/mad_internal.h > index e948540..24418cc 100644 > --- a/libibmad/src/mad_internal.h > +++ b/libibmad/src/mad_internal.h > @@ -39,6 +39,7 @@ > struct ibmad_port { > int port_id; /* file descriptor returned by umad_open() */ > int class_agents[MAX_CLASS]; /* class2agent mapper */ > + int timeout, retries; > }; > > extern struct ibmad_port *ibmp; > diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c > index 8c68cf9..3f3b7a2 100644 > --- a/libibmad/src/rpc.c > +++ b/libibmad/src/rpc.c > @@ -53,7 +53,7 @@ struct ibmad_port *ibmp = &mad_port; > static int iberrs; > > static int madrpc_retries = MAD_DEF_RETRIES; > -static int def_madrpc_timeout = MAD_DEF_TIMEOUT_MS; > +static int madrpc_timeout = MAD_DEF_TIMEOUT_MS; > static void *save_mad; > static int save_mad_len = 256; > > @@ -83,13 +83,23 @@ int madrpc_set_retries(int retries) > > int madrpc_set_timeout(int timeout) > { > - def_madrpc_timeout = timeout; > + madrpc_timeout = timeout; > return 0; > } > > +void mad_rpc_set_retries(struct ibmad_port *port, int retries) > +{ > + port->retries = retries; > +} > + > +void mad_rpc_set_timeout(struct ibmad_port *port, int timeout) > +{ > + port->timeout = timeout; > +} > + > int madrpc_def_timeout(void) > { > - return def_madrpc_timeout; > + return madrpc_timeout; > } > > int madrpc_portid(void) > @@ -104,15 +114,12 @@ int mad_rpc_portid(struct ibmad_port *srcport) > > static int > _do_madrpc(int port_id, void *sndbuf, void *rcvbuf, int agentid, int len, > - int timeout) > + int timeout, int max_retries) > { > uint32_t trid; /* only low 32 bits */ > int retries; > int length, status; > > - if (!timeout) > - timeout = def_madrpc_timeout; > - > if (ibdebug > 1) { > IBWARN(">>> sending: len %d pktsz %zu", len, umad_size() + len); > xdump(stderr, "send buf\n", sndbuf, umad_size() + len); > @@ -127,7 +134,7 @@ _do_madrpc(int port_id, void *sndbuf, void *rcvbuf, int agentid, int len, > trid = > (uint32_t) mad_get_field64(umad_get_mad(sndbuf), 0, IB_MAD_TRID_F); > > - for (retries = 0; retries < madrpc_retries; retries++) { > + for (retries = 0; retries < max_retries; retries++) { > if (retries) { > ERRS("retry %d (timeout %d ms)", retries, timeout); > } > @@ -171,6 +178,7 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport > { > int status, len; > uint8_t sndbuf[1024], rcvbuf[1024], *mad; > + int timeout, retries; > > len = 0; > memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); > @@ -178,9 +186,13 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport > if ((len = mad_build_pkt(sndbuf, rpc, dport, 0, payload)) < 0) > return 0; > > + timeout = rpc->timeout ? rpc->timeout : > + port->timeout ? port->timeout : madrpc_timeout; > + retries = port->retries ? port->retries : madrpc_retries; > + > if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, > port->class_agents[rpc->mgtclass], > - len, rpc->timeout)) < 0) { > + len, timeout, retries)) < 0) { > IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); > return 0; > } > @@ -209,6 +221,7 @@ void *mad_rpc_rmpp(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * > { > int status, len; > uint8_t sndbuf[1024], rcvbuf[1024], *mad; > + int timeout, retries; > > memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); > > @@ -217,9 +230,13 @@ void *mad_rpc_rmpp(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * > if ((len = mad_build_pkt(sndbuf, rpc, dport, rmpp, data)) < 0) > return 0; > > + timeout = rpc->timeout ? rpc->timeout : > + port->timeout ? port->timeout : madrpc_timeout; > + retries = port->retries ? port->retries : madrpc_retries; > + > if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, > port->class_agents[rpc->mgtclass], > - len, rpc->timeout)) < 0) { > + len, rpc->timeout, retries)) < 0) { > IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); > return 0; > } > -- > 1.6.1.2.319.gbd9e > -- Ira Weiny Math Programer/Computer Scientist Larence Livermore National Lab weiny2 at llnl.gov From or.gerlitz at gmail.com Wed Mar 11 10:02:14 2009 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Wed, 11 Mar 2009 19:02:14 +0200 Subject: ***SPAM*** Re: [ofa-general] tcp/rdma port unification patch in librdmacm In-Reply-To: <49B7D899.1030601@opengridcomputing.com> References: <49B17E15.4050406@opengridcomputing.com> <49B76CD5.705@voltaire.com> <49B7D3A0.5060307@opengridcomputing.com> <15ddcffd0903110819s3b83cec0vb87ee6114496858e@mail.gmail.com> <49B7D899.1030601@opengridcomputing.com> Message-ID: <15ddcffd0903111002o1db6c389s50d9ab0bc2bbd4b6@mail.gmail.com> > Depends on the 4-tuple. The demux done in the TOE hw uses  src addr/port and dst addr/port. is this somehow standardized or its being impl. dependent? Or. From hal.rosenstock at gmail.com Wed Mar 11 10:10:11 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 11 Mar 2009 12:10:11 -0500 Subject: ***SPAM*** Re: [ofa-general] ***SPAM*** [PATCH] infiniband-diags/mcm_rereg_test.c: Add missing mad_rpc_close_port call In-Reply-To: <20090311090337.ec293807.weiny2@llnl.gov> References: <20090311135320.GA29870@comcast.net> <20090311090337.ec293807.weiny2@llnl.gov> Message-ID: On Wed, Mar 11, 2009 at 11:03 AM, Ira Weiny wrote: > On Wed, 11 Mar 2009 08:53:20 -0500 > Hal Rosenstock wrote: > >> >> Signed-off-by: Hal Rosenstock >> >> --- >> diff --git a/infiniband-diags/src/mcm_rereg_test.c b/infiniband-diags/src/mcm_rereg_test.c >> index de5ea90..d087e4a 100644 >> --- a/infiniband-diags/src/mcm_rereg_test.c >> +++ b/infiniband-diags/src/mcm_rereg_test.c >> @@ -492,9 +492,13 @@ int main(int argc, char **argv) >>       mad = umad_get_mad(umad); >> >>       free(umad); >> +#if 1 >> +     mad_rpc_close_port(srcport); >> +#else >>       umad_unregister(port, agent); >>       umad_close_port(port); >>       umad_done(); >> +#endif > > Why use "#if 1" here?  Why not just replace the umad calls with > mad_rpc_close_port? Just consistency with what's been done in other places in that file. Other similar places are ifdef'd so I kept that. -- Hal > Ira > >> >>       return 0; >>  } >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http:// lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit http:// openib.org/mailman/listinfo/openib-general >> > > > -- > Ira Weiny > Math Programer/Computer Scientist > Larence Livermore National Lab > weiny2 at llnl.gov > - Show quoted text - > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From Don.Kerr at Sun.COM Wed Mar 11 10:16:15 2009 From: Don.Kerr at Sun.COM (Don Kerr) Date: Wed, 11 Mar 2009 13:16:15 -0400 Subject: [ofa-general] ibv_get_sysfs_path() Message-ID: <49B7F1DF.2040903@sun.com> Is the functionality provided by ibv_get_sysfs_path() and ibv_read_sysfs_file() defined in the IB Spec? Thanks -DON From swise at opengridcomputing.com Wed Mar 11 11:01:48 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 11 Mar 2009 13:01:48 -0500 Subject: [ofa-general] tcp/rdma port unification patch in librdmacm In-Reply-To: <15ddcffd0903111002o1db6c389s50d9ab0bc2bbd4b6@mail.gmail.com> References: <49B17E15.4050406@opengridcomputing.com> <49B76CD5.705@voltaire.com> <49B7D3A0.5060307@opengridcomputing.com> <15ddcffd0903110819s3b83cec0vb87ee6114496858e@mail.gmail.com> <49B7D899.1030601@opengridcomputing.com> <15ddcffd0903111002o1db6c389s50d9ab0bc2bbd4b6@mail.gmail.com> Message-ID: <49B7FC8C.7030602@opengridcomputing.com> Or Gerlitz wrote: >> Depends on the 4-tuple. The demux done in the TOE hw uses src addr/port and dst addr/port. >> > > is this somehow standardized or its being impl. dependent? > > Or. > This is standard TCP. The 4-tuple identifies the connection. From chien.tin.tung at intel.com Wed Mar 11 11:07:26 2009 From: chien.tin.tung at intel.com (Tung, Chien Tin) Date: Wed, 11 Mar 2009 11:07:26 -0700 Subject: [ofa-general] tcp/rdma port unification patch in librdmacm In-Reply-To: <49B7FC8C.7030602@opengridcomputing.com> References: <49B17E15.4050406@opengridcomputing.com> <49B76CD5.705@voltaire.com> <49B7D3A0.5060307@opengridcomputing.com> <15ddcffd0903110819s3b83cec0vb87ee6114496858e@mail.gmail.com> <49B7D899.1030601@opengridcomputing.com> <15ddcffd0903111002o1db6c389s50d9ab0bc2bbd4b6@mail.gmail.com> <49B7FC8C.7030602@opengridcomputing.com> Message-ID: <60BEFF3FBD4C6047B0F13F205CAFA383032E22A721@azsmsx501.amr.corp.intel.com> >Or Gerlitz wrote: >>> Depends on the 4-tuple. The demux done in the TOE hw uses >src addr/port and dst addr/port. >>> >> >> is this somehow standardized or its being impl. dependent? >> >> Or. >> > >This is standard TCP. The 4-tuple identifies the connection. > Steve, With the demuxing it would hurt performance, is this what we really want to do? On the other hand, I just checked our SLES 11 RC2 installation and the TCP port unification patch is there. If we can get a port reserved for iSER, we should be able to enable the unification patch by default and get out of this mess. What do you think? Chien From swise at opengridcomputing.com Wed Mar 11 11:11:26 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 11 Mar 2009 13:11:26 -0500 Subject: [ofa-general] tcp/rdma port unification patch in librdmacm In-Reply-To: <60BEFF3FBD4C6047B0F13F205CAFA383032E22A721@azsmsx501.amr.corp.intel.com> References: <49B17E15.4050406@opengridcomputing.com> <49B76CD5.705@voltaire.com> <49B7D3A0.5060307@opengridcomputing.com> <15ddcffd0903110819s3b83cec0vb87ee6114496858e@mail.gmail.com> <49B7D899.1030601@opengridcomputing.com> <15ddcffd0903111002o1db6c389s50d9ab0bc2bbd4b6@mail.gmail.com> <49B7FC8C.7030602@opengridcomputing.com> <60BEFF3FBD4C6047B0F13F205CAFA383032E22A721@azsmsx501.amr.corp.intel.com> Message-ID: <49B7FECE.3000308@opengridcomputing.com> Tung, Chien Tin wrote: >> Or Gerlitz wrote: >> >>>> Depends on the 4-tuple. The demux done in the TOE hw uses >>>> >> src addr/port and dst addr/port. >> >>>> >>>> >>> is this somehow standardized or its being impl. dependent? >>> >>> Or. >>> >>> >> This is standard TCP. The 4-tuple identifies the connection. >> >> > > Steve, > > With the demuxing it would hurt performance, is this what we really > want to do? > I don't know what you are talking about. TCP input demuxing is done in hardware for the iwarp devices. There is no performance issue here... > On the other hand, I just checked our SLES 11 RC2 installation > and the TCP port unification patch is there. If we can get a port > reserved for iSER, we should be able to enable the unification > patch by default and get out of this mess. What do you think? > > Chien Is it in RH too? Steve. From dotanba at gmail.com Wed Mar 11 12:42:32 2009 From: dotanba at gmail.com (Dotan Barak) Date: Wed, 11 Mar 2009 21:42:32 +0200 Subject: ***SPAM*** Re: [ofa-general] ibv_get_sysfs_path() In-Reply-To: <49B7F1DF.2040903@sun.com> References: <49B7F1DF.2040903@sun.com> Message-ID: <49B81428.2030204@gmail.com> Don Kerr wrote: > Is the functionality provided by ibv_get_sysfs_path() and > ibv_read_sysfs_file() defined in the IB Spec? I don't see any meaning for "sysfs" other than Linux/Unix. So the answer is "No", this is an implementation enhancement/feature. Dotan From Don.Kerr at Sun.COM Wed Mar 11 13:15:08 2009 From: Don.Kerr at Sun.COM (Don Kerr) Date: Wed, 11 Mar 2009 16:15:08 -0400 Subject: [ofa-general] ibv_get_sysfs_path() In-Reply-To: <49B81428.2030204@gmail.com> References: <49B7F1DF.2040903@sun.com> <49B81428.2030204@gmail.com> Message-ID: <49B81BCC.5060307@sun.com> On 03/11/09 15:42, Dotan Barak wrote: > Don Kerr wrote: >> Is the functionality provided by ibv_get_sysfs_path() and >> ibv_read_sysfs_file() defined in the IB Spec? > I don't see any meaning for "sysfs" other than Linux/Unix. > > So the answer is "No", this is an implementation enhancement/feature. OK. This is what I was thinking but wanted to be sure, thanks. It would be nice if the interface had a man page and stated as such. -DON > > Dotan From sean.hefty at intel.com Wed Mar 11 13:44:53 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 11 Mar 2009 13:44:53 -0700 Subject: [ofa-general] RE: merging madeye into mainline In-Reply-To: References: Message-ID: <34D0FA0BEDF644FAA31E979C8C81C3C5@amr.corp.intel.com> >Have you ever considered to push the madeye module (below) >into the kernel to ease with fabric debugging? I have tested >it now against Linus tree and it works fine. I hadn't really thought about it, but I don't have any objection to someone submitting it. There may be a better way of doing this if we want to include this upstream - for example, expose snooping capabilities to userspace. - Sean From sean.hefty at intel.com Wed Mar 11 13:54:30 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 11 Mar 2009 13:54:30 -0700 Subject: ***SPAM*** Re: [ofa-general] ***SPAM*** [PATCH] infiniband-diags/mcm_rereg_test.c: Add missing mad_rpc_close_port call In-Reply-To: References: <20090311135320.GA29870@comcast.net> <20090311090337.ec293807.weiny2@llnl.gov> Message-ID: <0DE999D9A91E4CCBA6714985A0211242@amr.corp.intel.com> >>> +#if 1 >>> +     mad_rpc_close_port(srcport); >>> +#else >>>       umad_unregister(port, agent); >>>       umad_close_port(port); >>>       umad_done(); >>> +#endif >> >> Why use "#if 1" here?  Why not just replace the umad calls with >> mad_rpc_close_port? > >Just consistency with what's been done in other places in that file. >Other similar places are ifdef'd so I kept that. My vote is to not add an #if 1 and keep dead code around. If #if 1 appears in other places, we should probably remove that dead code as well. From or.gerlitz at gmail.com Wed Mar 11 14:08:28 2009 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Wed, 11 Mar 2009 23:08:28 +0200 Subject: ***SPAM*** Re: [ofa-general] tcp/rdma port unification patch in librdmacm In-Reply-To: <49B7FC8C.7030602@opengridcomputing.com> References: <49B17E15.4050406@opengridcomputing.com> <49B76CD5.705@voltaire.com> <49B7D3A0.5060307@opengridcomputing.com> <15ddcffd0903110819s3b83cec0vb87ee6114496858e@mail.gmail.com> <49B7D899.1030601@opengridcomputing.com> <15ddcffd0903111002o1db6c389s50d9ab0bc2bbd4b6@mail.gmail.com> <49B7FC8C.7030602@opengridcomputing.com> Message-ID: <15ddcffd0903111408x5530dabbn54f3c9c4642f31f4@mail.gmail.com> > This is standard TCP.   The 4-tuple identifies the connection. I understand that standard TCP demuxing is based on the four-tuple, but I didn't think we were talking TCP but rather on the NIC L2 MAC HW dispatcher that has to decide to which input queue (QP if you want) it should place a frame that reached the card, so this dispatcher has the L2, L3, L4 (and maybe also the DDP) headers and I understand that the Chelsio card does so based on the L3 and L4 info. Or. From sean.hefty at intel.com Wed Mar 11 14:21:18 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 11 Mar 2009 14:21:18 -0700 Subject: [ofa-general] RE: tcp/rdma port unification patch in librdmacm In-Reply-To: <49B7D41C.4030707@opengridcomputing.com> References: <49B17E15.4050406@opengridcomputing.com> <811553D6D34A4E6F9AB686CA2E441D35@amr.corp.intel.com> <49B61C5D.6060900@Voltaire.com> <3281521B4F4A415A846A9F2E8AF9C21F@amr.corp.intel.com> <49B767DB.2050700@voltaire.com> <49B7D41C.4030707@opengridcomputing.com> Message-ID: <23854445101749EA8CE12A1BF788F6F6@amr.corp.intel.com> >Or maybe map PS_TCP port space to the be the unified space, and have a >new PS_RDMA or PS_IB port space that doesn't? Hmm... I like this concept if a userspace work-around is needed. From weiny2 at llnl.gov Wed Mar 11 14:44:03 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 11 Mar 2009 14:44:03 -0700 Subject: [ofa-general] [PATCH] libibmad: bump library interface version Message-ID: <20090311144403.5524d85c.weiny2@llnl.gov> From: Ira Weiny Date: Wed, 11 Mar 2009 10:44:28 -0700 Subject: [PATCH] libibmad: bump library interface version There has been enough interface changes to warrant a new version. Signed-off-by: Ira Weiny --- libibmad/libibmad.ver | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/libibmad/libibmad.ver b/libibmad/libibmad.ver index 7a96f8d..1d405d4 100644 --- a/libibmad/libibmad.ver +++ b/libibmad/libibmad.ver @@ -6,4 +6,4 @@ # API_REV - advance on any added API # RUNNING_REV - advance any change to the vendor files # AGE - number of backward versions the API still supports -LIBVERSION=2:1:0 +LIBVERSION=3:0:1 -- 1.5.4.5 From weiny2 at llnl.gov Wed Mar 11 14:44:04 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 11 Mar 2009 14:44:04 -0700 Subject: [ofa-general] [PATCH] Update mad formatting functions. Message-ID: <20090311144404.bf15ba8b.weiny2@llnl.gov> From: Ira Weiny Date: Wed, 11 Mar 2009 10:45:25 -0700 Subject: [PATCH] Update mad formatting functions. Add mad_snprintf w/ man page Add mad_fprintf w/ man page Add comments to document current functions. Rename parameters to avoid confusion with other functions which take "buf" Mark mad_print_field as deprecated Signed-off-by: Ira Weiny --- libibmad/Makefile.am | 2 + libibmad/include/infiniband/mad.h | 28 +++- libibmad/man/mad_fprintf.3 | 82 ++++++++++ libibmad/man/mad_snprintf.3 | 2 + libibmad/src/fields.c | 319 ++++++++++++++++++++++++++++++++++++- libibmad/src/libibmad.map | 2 + 6 files changed, 428 insertions(+), 7 deletions(-) create mode 100644 libibmad/man/mad_fprintf.3 create mode 100644 libibmad/man/mad_snprintf.3 diff --git a/libibmad/Makefile.am b/libibmad/Makefile.am index 4f3ba98..da32899 100644 --- a/libibmad/Makefile.am +++ b/libibmad/Makefile.am @@ -5,6 +5,8 @@ INCLUDES = -I$(srcdir)/include -I$(includedir) lib_LTLIBRARIES = libibmad.la +man_MANS = man/mad_fprintf.3 man/mad_snprintf.3 + libibmad_la_CFLAGS = -Wall if HAVE_LD_VERSION_SCRIPT diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h index 064cbb7..897b91b 100644 --- a/libibmad/include/infiniband/mad.h +++ b/libibmad/include/infiniband/mad.h @@ -719,9 +719,31 @@ MAD_EXPORT void mad_set_array(void *buf, int base_offs, enum MAD_FIELDS field, v MAD_EXPORT void mad_get_array(void *buf, int base_offs, enum MAD_FIELDS field, void *val); MAD_EXPORT void mad_decode_field(uint8_t * buf, enum MAD_FIELDS field, void *val); MAD_EXPORT void mad_encode_field(uint8_t * buf, enum MAD_FIELDS field, void *val); -MAD_EXPORT int mad_print_field(enum MAD_FIELDS field, const char *name, void *val); -MAD_EXPORT char *mad_dump_field(enum MAD_FIELDS field, char *buf, int bufsz, void *val); -MAD_EXPORT char *mad_dump_val(enum MAD_FIELDS field, char *buf, int bufsz, void *val); +MAD_EXPORT int mad_print_field(enum MAD_FIELDS field, const char *name, void *val) + DEPRECATED; + +/** + * The following functions print fields to "s" in various ways + * + * mad_dump_[val|field] take a value "val" and use "field" to format it + * + * mad_snprint_field takes a data buffer "buf" and uses field to extract and + * format it. + * + * RETURN "s" or NULL on failure + */ +MAD_EXPORT char *mad_dump_field(enum MAD_FIELDS field, char *s, int n, void *val); + /* outputs string ":........" */ +MAD_EXPORT char *mad_dump_val(enum MAD_FIELDS field, char *s, int n, void *val); + /* outputs string "" */ + +/** + * printf functions + * input's "standard" printf parameters except for "buf" which is a mad buffer + * return the number of actual chars written to "s" or "stream" + */ +MAD_EXPORT int mad_snprintf(char *s, size_t n, uint8_t *buf, const char *format, ...); +MAD_EXPORT int mad_fprintf(FILE *stream, uint8_t *buf, const char *format, ...); /* mad.c */ MAD_EXPORT void *mad_encode(void *buf, ib_rpc_t * rpc, ib_dr_path_t * drpath, diff --git a/libibmad/man/mad_fprintf.3 b/libibmad/man/mad_fprintf.3 new file mode 100644 index 0000000..e69bd44 --- /dev/null +++ b/libibmad/man/mad_fprintf.3 @@ -0,0 +1,82 @@ +.\" -*- nroff -*- +.\" +.TH MAD_FPRINTF 3 "Feb 26, 2009" "OpenIB" "OpenIB Programmer\'s Manual" +.SH "NAME" +mad_fprintf, mad_snprintf \- formatted output conversion for mad packets +.SH "SYNOPSIS" +.nf +.B #include +.sp +.BI "MAD_EXPORT int mad_snprintf(char " "*s" ", size_t "n ", uint8_t " "*buf" ", const char " "*format" ", ...); +.BI "MAD_EXPORT int mad_fprintf(FILE " "*stream" ", uint8_t " "*buf" ", const char " "*format" ", ...); +.fi +.SH "DESCRIPTION" +Similar to the printf family of functions. The exception being they do +.B not +accept all conversion specifiers and they accept a "buf" parameter which +represents a mad data buffer. This buffer is used to extract and print fields +as specified with the +.B %F +format specifier. +.PP +The following conversion specifiers are +.B not +supported. +.B e, E, f, g, G, a, A, C, S, m +and +.B n +.PP +.B F +The %F specifier is used to print out fields decoded from the "buf" data +buffer. +.I enum MAD_FIELDS\fR +values should be used to specify the field to be decoded. +.PP +.SH "EXAMPLES" +.nf +char portinfo[64]; +void *pi = portinfo; +.PP +if (!smp_query(pi, portid, IB_ATTR_PORT_INFO, portnum, timeout)) +.in +8 + return -1; +.in -16 +.PP +mad_fprintf(stdout, pi, "Port info (%s):\\n" +.in +16 +" %-10s: %F\\n" +" %-10s: %F\\n" +" %-10s: %F\\n" +" %-10s: %F\\n" +" %-10s: %F\\n" +" %-10s: %F\\n", +portid2str(portid), +"LID", IB_PORT_LID_F, +"LMC", IB_PORT_LMC_F, +"state", IB_PORT_STATE_F, +"physstate", IB_PORT_PHYS_STATE_F, +"linkwidth", IB_PORT_LINK_WIDTH_ACTIVE_F, +"linkspeed", IB_PORT_LINK_SPEED_ACTIVE_F +); +.in -16 +.PP +Results in the output. +.PP +Port info (DR path slid 0; dlid 0; 0,1,14): +.in +3 +LID : 0x0016 +LMC : 0 +state : Active +physstate : LinkUp +linkwidth : 4X +linkspeed : 5.0 Gbps +.in -3 + +.SH "RETURN VALUE" +.B return the number of characters printed. + +.SH "SEE ALSO" +.BR printf (3) +.SH "AUTHOR" +.TP +Ira Weiny diff --git a/libibmad/man/mad_snprintf.3 b/libibmad/man/mad_snprintf.3 new file mode 100644 index 0000000..c004ab9 --- /dev/null +++ b/libibmad/man/mad_snprintf.3 @@ -0,0 +1,2 @@ +.TH MAD_SNPRINTF 3 "Feb 26, 2009" "OpenIB" "OpenIB Programmer\'s Manual" +.so man3/mad_fprintf.3 diff --git a/libibmad/src/fields.c b/libibmad/src/fields.c index 19c8fc1..acb1180 100644 --- a/libibmad/src/fields.c +++ b/libibmad/src/fields.c @@ -38,7 +38,9 @@ #include #include +#define _GNU_SOURCE #include +#include #include @@ -442,6 +444,9 @@ static const ib_field_t ib_mad_f[] = { }; +#define MAD_FIELD_MAX_BYTE_LEN (256) + /* currently "Vendor2Data" increased to the next power of 2 */ + static void _set_field64(void *buf, int base_offs, const ib_field_t * f, uint64_t val) { @@ -666,6 +671,7 @@ static int _mad_print_field(const ib_field_t * f, const char *name, void *val, valsz ? valsz : ALIGN(f->bitlen, 8) / 8); } +/* This function is deprecated use mad_snprint_field or mad_dump_* instead */ int mad_print_field(enum MAD_FIELDS field, const char *name, void *val) { if (field <= IB_NO_FIELD || field >= IB_FIELD_LAST_) @@ -673,16 +679,321 @@ int mad_print_field(enum MAD_FIELDS field, const char *name, void *val) return _mad_print_field(ib_mad_f + field, name, val, 0); } -char *mad_dump_field(enum MAD_FIELDS field, char *buf, int bufsz, void *val) +char *mad_dump_field(enum MAD_FIELDS field, char *s, int n, void *val) { if (field <= IB_NO_FIELD || field >= IB_FIELD_LAST_) return 0; - return _mad_dump_field(ib_mad_f + field, 0, buf, bufsz, val); + return _mad_dump_field(ib_mad_f + field, 0, s, n, val); } -char *mad_dump_val(enum MAD_FIELDS field, char *buf, int bufsz, void *val) +char *mad_dump_val(enum MAD_FIELDS field, char *s, int n, void *val) { if (field <= IB_NO_FIELD || field >= IB_FIELD_LAST_) return 0; - return _mad_dump_val(ib_mad_f + field, buf, bufsz, val); + return _mad_dump_val(ib_mad_f + field, s, n, val); } + +#define ZEROPAD 1 /* pad with zero */ +#define SIGN 2 /* unsigned/signed long */ +#define PLUS 4 /* show plus */ +#define SPACE 8 /* space if plus */ +#define LEFT 16 /* left justified */ +#define SPECIAL 32 /* 0x */ +#define LARGE 64 /* use 'ABCDEF' instead of 'abcdef' */ + +static char * number(char * str, size_t n, int *rc, + unsigned long long num, int base, int size, + int precision, int type) +{ + char c,sign,tmp[66]; + const char *digits="0123456789abcdefghijklmnopqrstuvwxyz"; + int i; + +/* Macro allows for checking length + * remove 1 to allow for \0 char */ +#define WRITE_CHAR_RET(c) do { \ + *str++ = c; \ + if (++(*rc) >= (n-1)) { \ + return (str); \ + } \ +} while(0) + + if (type & LARGE) + digits = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ"; + if (type & LEFT) + type &= ~ZEROPAD; + if (base < 2 || base > 36) + return 0; + c = (type & ZEROPAD) ? '0' : ' '; + sign = 0; + if (type & SIGN) { + if ((signed long long)num < 0) { + sign = '-'; + num = - (signed long long)num; + size--; + } else if (type & PLUS) { + sign = '+'; + size--; + } else if (type & SPACE) { + sign = ' '; + size--; + } + } + if (type & SPECIAL) { + if (base == 16) + size -= 2; + else if (base == 8) + size--; + } + i = 0; + if (num == 0) + tmp[i++]='0'; + else { + while (num >= base) { + tmp[i++] = digits[num % base]; + num /= base; + } + tmp[i++] = digits[num]; + } + if (i > precision) + precision = i; + size -= precision; + if (!(type&(ZEROPAD+LEFT))) + while(size-->0) + //*str++ = ' '; + WRITE_CHAR_RET(' '); + if (sign) + //*str++ = sign; + WRITE_CHAR_RET(sign); + if (type & SPECIAL) { + if (base==8) + WRITE_CHAR_RET('0'); + else if (base==16) { + WRITE_CHAR_RET('0'); + WRITE_CHAR_RET(digits[33]); + } + } + if (!(type & LEFT)) + while (size-- > 0) + WRITE_CHAR_RET(c); + while (i < precision--) + WRITE_CHAR_RET('0'); + while (i-- > 0) + WRITE_CHAR_RET(tmp[i]); + while (size-- > 0) + WRITE_CHAR_RET(' '); + return str; +} + +static int mad_vsnprintf(char *s, size_t n, void *buf, const char *fmt, va_list args) +{ + int rc = 0; + int len; + unsigned long long num; + int i, base; + char *str; + +/* Macros allows for bounding length of print to provided buffer + * remove 1 to allow for \0 char */ +#define WRITE_CHAR(c) do { \ + *str++ = c; \ + if (++rc >= (n-1)) { \ + goto max_len_hit; \ + } \ +} while(0) +#define WRITE_STR(STR) do { \ + const char *ls = STR; \ + len = strlen(ls); \ + if (precision > 0 && len > precision) \ + len = precision; \ + if (!(flags & LEFT)) \ + while (len < field_width--) \ + WRITE_CHAR(' '); \ + for (i = 0; i < len; ++i) \ + WRITE_CHAR(*ls++); \ + while (len < field_width--) \ + WRITE_CHAR(' '); \ +} while (0); + + int flags; + int field_width; + int precision; + int qualifier; + + for (str=s ; *fmt ; ++fmt) { + if (*fmt != '%') { + //*str++ = *fmt; + WRITE_CHAR(*fmt); + continue; + } + + /* process flags */ + flags = 0; +repeat: + ++fmt; /* this also skips first '%' */ + switch (*fmt) { + case '-': flags |= LEFT; goto repeat; + case '+': flags |= PLUS; goto repeat; + case ' ': flags |= SPACE; goto repeat; + case '#': flags |= SPECIAL; goto repeat; + case '0': flags |= ZEROPAD; goto repeat; + } + + /* get field width */ + field_width = -1; + if ('0' <= *fmt && *fmt <= '9') { + int c = 0; + for (field_width = 0; '0' <= (c = *fmt) && c <= '9'; ++fmt) + field_width = field_width*10 + c - '0'; + } else if (*fmt == '*') { + ++fmt; + /* it's the next argument */ + field_width = va_arg(args, int); + if (field_width < 0) { + field_width = -field_width; + flags |= LEFT; + } + } + + /* get the precision */ + precision = -1; + if (*fmt == '.') { + ++fmt; + if ('0' <= *fmt && *fmt <= '9') { + int c = 0; + for (precision = 0; '0' <= (c = *fmt) && c <= '9'; ++fmt) + precision = precision*10 + c - '0'; + } else if (*fmt == '*') { + ++fmt; + /* it's the next argument */ + precision = va_arg(args, int); + } + if (precision < 0) + precision = 0; + } + + /* get the conversion qualifier */ + qualifier = -1; + if (*fmt == 'h' || *fmt == 'l' || *fmt == 'L' || *fmt =='z') { + qualifier = *fmt; + ++fmt; + } + + /* default base */ + base = 10; + + switch (*fmt) { + case 'F': + { + char s[256]; + uint8_t val[MAD_FIELD_MAX_BYTE_LEN]; + int field = va_arg(args, int); + const ib_field_t *f = ib_mad_f + field; + + if (field <= IB_NO_FIELD || field >= IB_FIELD_LAST_) + continue; + + mad_decode_field(buf, field, val); + f->def_dump_fn(s, n, val, ALIGN(f->bitlen, 8) / 8); + WRITE_STR(s); + continue; + } + case 'c': + if (!(flags & LEFT)) + while (--field_width > 0) + WRITE_CHAR(' '); + WRITE_CHAR((unsigned char) va_arg(args, int)); + while (--field_width > 0) + WRITE_CHAR(' '); + continue; + + case 's': + { + const char *string = va_arg(args, char *); + if (!string) + string = ""; + + WRITE_STR(string); + continue; + } + + case '%': + WRITE_CHAR('%'); + continue; + + /* integer number formats - set up the flags and "break" */ + case 'o': + base = 8; + break; + + case 'p': + case 'X': + flags |= LARGE; + case 'x': + base = 16; + break; + + case 'd': + case 'i': + flags |= SIGN; + case 'u': + break; + + default: + WRITE_CHAR('%'); + if (*fmt) + WRITE_CHAR(*fmt); + else + --fmt; + continue; + } + if (qualifier == 'l') { + num = va_arg(args, unsigned long); + if (flags & SIGN) + num = (signed long) num; + } else if (qualifier == 'z') { + num = va_arg(args, size_t); + } else if (qualifier == 'h') { + num = (unsigned short) va_arg(args, int); + if (flags & SIGN) + num = (signed short) num; + } else { + num = va_arg(args, unsigned int); + if (flags & SIGN) + num = (signed int) num; + } + str = number(str, n-rc, &rc, num, base, field_width, precision, flags); + if (rc <= 1) + break; + } +max_len_hit: + *str = '\0'; + return str-s; +} + +int mad_snprintf(char *s, size_t n, uint8_t *buf, const char *format, ...) +{ + va_list args; + int i; + + va_start(args, format); + i = mad_vsnprintf(s, n, buf, format, args); + va_end(args); + return (i); +} + +int mad_fprintf(FILE *stream, uint8_t *buf, const char *format, ...) +{ + char str_buf[1024]; + va_list args; + int i,j; + + va_start(args, format); + i = mad_vsnprintf(str_buf, 1024, buf, format, args); + va_end(args); + j = fprintf(stream, "%s", str_buf); + if (i != j) + IBWARN("mad_vsnprintf and fprintf don't match???\n"); + return (i); +} + diff --git a/libibmad/src/libibmad.map b/libibmad/src/libibmad.map index 0be7a92..2265b12 100644 --- a/libibmad/src/libibmad.map +++ b/libibmad/src/libibmad.map @@ -4,6 +4,8 @@ IBMAD_1.3 { mad_dump_field; mad_dump_val; mad_print_field; + mad_snprintf; + mad_fprintf; mad_dump_array; mad_dump_bitfield; mad_dump_hex; -- 1.5.4.5 From weiny2 at llnl.gov Wed Mar 11 14:44:06 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 11 Mar 2009 14:44:06 -0700 Subject: [ofa-general] [PATCH] NOT For inclusion in master: Simple example of mad_fprintf in action. Message-ID: <20090311144406.8cf92ba6.weiny2@llnl.gov> Just an example of the use of mad_fprintf I don't think this needs to go into master. Ira From: Ira Weiny Date: Wed, 11 Mar 2009 10:46:59 -0700 Subject: [PATCH] Simple example of mad_fprintf in action. Signed-off-by: Ira Weiny --- infiniband-diags/src/ibnetdiscover.c | 17 +++++++++++++++++ 1 files changed, 17 insertions(+), 0 deletions(-) diff --git a/infiniband-diags/src/ibnetdiscover.c b/infiniband-diags/src/ibnetdiscover.c index 25c1f7f..e61f59d 100644 --- a/infiniband-diags/src/ibnetdiscover.c +++ b/infiniband-diags/src/ibnetdiscover.c @@ -186,6 +186,23 @@ get_node(Node *node, Port *port, ib_portid_t *portid) return -1; decode_port_info(pi, port); + if (ibdebug || ibverbose) + mad_fprintf(stdout, pi, "Port info (%s):\n" + " %-10s: %F\n" + " %-10s: %F\n" + " %-10s: %F\n" + " %-10s: %F\n" + " %-10s: %F\n" + " %-10s: %F\n", + portid2str(portid), + "LID", IB_PORT_LID_F, + "LMC", IB_PORT_LMC_F, + "state", IB_PORT_STATE_F, + "physstate", IB_PORT_PHYS_STATE_F, + "linkwidth", IB_PORT_LINK_WIDTH_ACTIVE_F, + "linkspeed", IB_PORT_LINK_SPEED_ACTIVE_F + ); + if (node->type != SWITCH_NODE) return 0; -- 1.5.4.5 From rdreier at cisco.com Wed Mar 11 15:50:29 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 11 Mar 2009 15:50:29 -0700 Subject: [ofa-general][PATCH 1/2 V2] mlx4_core: Don't perform SET_PORT command for Eth port from core In-Reply-To: <49B3DF9C.4080803@mellanox.co.il> (Yevgeny Petrilin's message of "Sun, 08 Mar 2009 17:09:16 +0200") References: <49B3DF9C.4080803@mellanox.co.il> Message-ID: thanks, applied From sean.hefty at intel.com Wed Mar 11 15:52:00 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 11 Mar 2009 15:52:00 -0700 Subject: [ofa-general] [PATCH] libibmad: bump library interface version In-Reply-To: <20090311144403.5524d85c.weiny2@llnl.gov> References: <20090311144403.5524d85c.weiny2@llnl.gov> Message-ID: > There has been enough interface changes to warrant a new version. If we're bumping the version anyway, is what's there now for retries/timeouts the best thing longer term? (I haven't tracked each change in detail, so I don't know.) - Sean From weiny2 at llnl.gov Wed Mar 11 16:21:59 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 11 Mar 2009 16:21:59 -0700 Subject: [ofa-general] [PATCH] libibmad: bump library interface version In-Reply-To: References: <20090311144403.5524d85c.weiny2@llnl.gov> Message-ID: <20090311162159.4e0e9571.weiny2@llnl.gov> On Wed, 11 Mar 2009 15:52:00 -0700 "Sean Hefty" wrote: > > There has been enough interface changes to warrant a new version. > > If we're bumping the version anyway, is what's there now for retries/timeouts > the best thing longer term? (I haven't tracked each change in detail, so I > don't know.) I believe what Sasha sent out last is good enough for the transition. Ira > > - Sean > -- Ira Weiny Math Programer/Computer Scientist Larence Livermore National Lab weiny2 at llnl.gov From arlin.r.davis at intel.com Wed Mar 11 16:28:09 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Wed, 11 Mar 2009 16:28:09 -0700 Subject: [ofa-general] Bandwidth of performance with multirail IB In-Reply-To: <49A49458.9070003@cs.anu.edu.au> References: <20090223211155.730AFE28137@openfabrics.org> <49A378BC.5010806@cs.anu.edu.au> <200902240941.58634.cap@nsc.liu.se> <49A49458.9070003@cs.anu.edu.au> Message-ID: Jie, >>> The peak bandwidth I can get is ~ 1.3 GB/s (not >bi-directional), which >>> is almost the same as single rail connections. >>> That is correct 10Gb/s each port for a max of 20Gb/s. >> >> Assuming you have a 2.5 GT/s pci-express x8 that speed is a >result of the bus >> not being able to keep up with the HCA. Since the bus is >holding even a >> single DDR IB port back you see no improvement with two ports. >> >> >I do connect HCA in a 16x pci-e slot on each node. >However, I am trying to drive 2 ports simultaneously. > >The workstation i am using is Sun Ultra 24, >and the HCA is Mellanox ConnectX MHGH28-XTC. >The data for the HCA and Ultra 24 is > >MHGH28-XTC >IB ports: Dual Copper 4X 20Gb/s >Host Bus: PCIe 2.0 2.5GT/s Peter is right. Your maximum speed given this configuration is 20Gb/s (40Gb/s bi-directional) since you have an 8x adapter in a 2.5GT/s bus. It doesn't matter that you have it in a 16x slot, your adapter is only 8x 2.5GT/s. -arlin From phillipwils at gmail.com Wed Mar 11 18:18:57 2009 From: phillipwils at gmail.com (Phillip Wilson) Date: Wed, 11 Mar 2009 18:18:57 -0700 Subject: [ofa-general] ***SPAM*** ib_mthca 0000:06:00.0: Catastrophic error detected: internal error Message-ID: <6e4f44220903111818r2d60baaes7bf485bea5641060@mail.gmail.com> I looked through the ofa-general mail archives and found this issue in the November 6 through 10, 2008 archives. According to the mail thread, the issue was fix by upgrading the firmware. I have the latest posted 1.2.0 firmware, but I am running into the "ib_mthca: Catastrophic error detected: internal error" issue. The cards are reset when this issue occurs and the LIDs ( sm_lid, port_lid) are reset to 0. #> ibv_devinfo hca_id: mthca2 fw_ver: 1.2.0 node_guid: 0019:bbff:fff8:8184 sys_image_guid: 0019:bbff:fff8:8187 vendor_id: 0x02c9 vendor_part_id: 25204 hw_ver: 0xA0 board_id: HP_0010000001 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 2048 (4) active_mtu: 512 (2) sm_lid: 1 port_lid: 1 port_lmc: 0x00 hca_id: mthca1 fw_ver: 1.2.0 node_guid: 0019:bbff:fff7:3c40 sys_image_guid: 0019:bbff:fff7:3c43 vendor_id: 0x02c9 vendor_part_id: 25204 hw_ver: 0xA0 board_id: HP_0010000001 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 2048 (4) active_mtu: 512 (2) sm_lid: 0 port_lid: 0 port_lmc: 0x00 hca_id: mthca0 fw_ver: 1.2.0 node_guid: 0019:bbff:fff7:4b10 sys_image_guid: 0019:bbff:fff7:4b13 vendor_id: 0x02c9 vendor_part_id: 25204 hw_ver: 0xA0 board_id: HP_0010000001 phys_port_cnt: 1 port: 1 state: PORT_DOWN (1) max_mtu: 2048 (4) active_mtu: 512 (2) sm_lid: 0 port_lid: 0 port_lmc: 0x00 #> dmesg M95700A6) rev 2100 PHY(serdes)] (PCIX:66MHz:64-bit) 1000Base-SX Ethernet 00:1b:78:7c:11:b7 eth3: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] WireSpeed[0] TSOcap[1] eth3: dma_rwctrl[769f0000] dma_mask[64-bit] Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx Probing IDE interface ide0... Probing IDE interface ide1... Probing IDE interface ide2... Probing IDE interface ide3... Loading iSCSI transport class v2.0-724. QLogic Fibre Channel HBA Driver iscsi: registered transport (qla4xxx) QLogic iSCSI HBA Driver Emulex LightPulse Fibre Channel SCSI driver 8.2.2 Copyright(c) 2004-2007 Emulex. All rights reserved. Driver 'sd' needs updating - please use bus_type methods Driver 'sr' needs updating - please use bus_type methods Fusion MPT base driver 3.04.06 Copyright (c) 1999-2007 LSI Corporation Fusion MPT SPI Host driver 3.04.06 Fusion MPT FC Host driver 3.04.06 Fusion MPT SAS Host driver 3.04.06 GSI 38 (level, low) -> CPU 10 (0x0a00) vector 61 ACPI: PCI Interrupt 0000:02:01.0[A] -> GSI 38 (level, low) -> IRQ 61 mptbase: ioc0: Initiating bringup ioc0: LSISAS1068 B0: Capabilities={Initiator} scsi0 : ioc0: LSISAS1068 B0, FwRev=01172100h, Ports=1, MaxQ=163, IRQ=61 scsi 0:0:0:0: Direct-Access HP DG146ABAB4 HPD5 PQ: 0 ANSI: 5 sd 0:0:0:0: [sda] 286749488 512-byte hardware sectors (146816 MB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Mode Sense: db 00 10 08 sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA sd 0:0:0:0: [sda] 286749488 512-byte hardware sectors (146816 MB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Mode Sense: db 00 10 08 sd 0:0:0:0: [sda] Write cache: disabled, read cache: enabled, supports DPO and FUA sda: sda1 sda2 sda3 sd 0:0:0:0: [sda] Attached SCSI disk sd 0:0:0:0: Attached scsi generic sg0 type 0 GSI 19 (level, low) -> CPU 11 (0x0b00) vector 62 ACPI: PCI Interrupt 0000:00:02.2[C] -> GSI 19 (level, low) -> IRQ 62 ehci_hcd 0000:00:02.2: EHCI Host Controller ehci_hcd 0000:00:02.2: new USB bus registered, assigned bus number 1 ehci_hcd 0000:00:02.2: irq 62, io mem 0x88030000 ehci_hcd 0000:00:02.2: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 usb usb1: configuration #1 chosen from 1 choice hub 1-0:1.0: USB hub found hub 1-0:1.0: 5 ports detected ohci_hcd: 2006 August 04 USB 1.1 'Open' Host Controller (OHCI) Driver GSI 17 (level, low) -> CPU 12 (0x0c00) vector 63 ACPI: PCI Interrupt 0000:00:02.0[A] -> GSI 17 (level, low) -> IRQ 63 ohci_hcd 0000:00:02.0: OHCI Host Controller ohci_hcd 0000:00:02.0: new USB bus registered, assigned bus number 2 ohci_hcd 0000:00:02.0: irq 63, io mem 0x88032000 usb usb2: configuration #1 chosen from 1 choice hub 2-0:1.0: USB hub found hub 2-0:1.0: 3 ports detected GSI 18 (level, low) -> CPU 13 (0x0d00) vector 64 ACPI: PCI Interrupt 0000:00:02.1[B] -> GSI 18 (level, low) -> IRQ 64 ohci_hcd 0000:00:02.1: OHCI Host Controller ohci_hcd 0000:00:02.1: new USB bus registered, assigned bus number 3 ohci_hcd 0000:00:02.1: irq 64, io mem 0x88031000 usb usb3: configuration #1 chosen from 1 choice hub 3-0:1.0: USB hub found hub 3-0:1.0: 2 ports detected USB Universal Host Controller Interface driver v3.0 Initializing USB Mass Storage driver... usb 2-1: new full speed USB device using ohci_hcd and address 2 usb 2-1: configuration #1 chosen from 1 choice usbcore: registered new interface driver usb-storage USB Mass Storage support registered. mice: PS/2 mouse device common for all mice md: raid0 personality registered for level 0 device-mapper: ioctl: 4.12.0-ioctl (2007-10-02) initialised: dm-devel at redhat.com ib_mthca: Mellanox InfiniBand HCA driver v0.08 (February 14, 2006) ib_mthca: Initializing 0000:04:00.0 ACPI: PCI Interrupt 0000:04:00.0[A] -> GSI 49 (level, low) -> IRQ 50 ib_mthca: Initializing 0000:06:00.0 ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 56 (level, low) -> IRQ 52 ib_mthca: Initializing 0000:08:00.0 ACPI: PCI Interrupt 0000:08:00.0[A] -> GSI 63 (level, low) -> IRQ 54 iscsi: registered transport (iser) EFI Variables Facility v0.08 2004-May-17 input: HP Virtual Management Device as /devices/pci0000:00/0000:00:02.0/usb2/2-1/2-1:1.0/input/input0 input: USB HID v1.11 Keyboard [HP Virtual Management Device] on usb-0000:00:02.0-1 input: HP Virtual Management Device as /devices/pci0000:00/0000:00:02.0/usb2/2-1/2-1:1.1/input/input1 input: USB HID v1.01 Mouse [HP Virtual Management Device] on usb-0000:00:02.0-1 usbcore: registered new interface driver usbhid drivers/hid/usbhid/hid-core.c: v2.6:USB HID core driver TCP cubic registered NET: Registered protocol family 1 NET: Registered protocol family 17 md: Autodetecting RAID arrays. md: Scanned 0 and added 0 devices. md: autorun ... md: ... autorun DONE. RAMDISK: Compressed image found at block 0 EXT2-fs warning: mounting unchecked fs, running e2fsck is recommended VFS: Mounted root (ext2 filesystem). Freeing unused kernel memory: 1744kB freed Intel(R) Gigabit Ethernet Network Driver - version 1.2.44.9 Copyright (c) 2007-2008 Intel Corporation. user_mad: process infiniband did not enable P_Key index support. user_mad: Documentation/infiniband/user_mad.txt has info on the new ABI. user_mad: process infiniband did not enable P_Key index support. user_mad: Documentation/infiniband/user_mad.txt has info on the new ABI. user_mad: process infiniband did not enable P_Key index support. user_mad: Documentation/infiniband/user_mad.txt has info on the new ABI. user_mad: process infiniband did not enable P_Key index support. user_mad: Documentation/infiniband/user_mad.txt has info on the new ABI. ib_mthca 0000:04:00.0: Catastrophic error detected: internal error ib_mthca 0000:04:00.0: buf[00]: 0012f6f8 ib_mthca 0000:04:00.0: buf[01]: 00000000 ib_mthca 0000:04:00.0: buf[02]: 00000000 ib_mthca 0000:04:00.0: buf[03]: 00000000 ib_mthca 0000:04:00.0: buf[04]: 00000000 ib_mthca 0000:04:00.0: buf[05]: 0012f6dc ib_mthca 0000:04:00.0: buf[06]: 001868f4 ib_mthca 0000:04:00.0: buf[07]: 00000000 ib_mthca 0000:04:00.0: buf[08]: 00000000 ib_mthca 0000:04:00.0: buf[09]: 00000000 ib_mthca 0000:04:00.0: buf[0a]: 00000000 ib_mthca 0000:04:00.0: buf[0b]: 00000000 ib_mthca 0000:04:00.0: buf[0c]: 00000000 ib_mthca 0000:04:00.0: buf[0d]: 00000000 ib_mthca 0000:04:00.0: buf[0e]: 00000000 ib_mthca 0000:04:00.0: buf[0f]: 00000000 ACPI: PCI interrupt for device 0000:04:00.0 disabled ib_mthca: Initializing 0000:04:00.0 ACPI: PCI Interrupt 0000:04:00.0[A] -> GSI 49 (level, low) -> IRQ 50 user_mad: process infiniband did not enable P_Key index support. user_mad: Documentation/infiniband/user_mad.txt has info on the new ABI. ib_mthca 0000:06:00.0: Catastrophic error detected: internal error ib_mthca 0000:06:00.0: buf[00]: 0012f6f8 ib_mthca 0000:06:00.0: buf[01]: 00000000 ib_mthca 0000:06:00.0: buf[02]: 00000000 ib_mthca 0000:06:00.0: buf[03]: 00000000 ib_mthca 0000:06:00.0: buf[04]: 00000000 ib_mthca 0000:06:00.0: buf[05]: 0012f6dc ib_mthca 0000:06:00.0: buf[06]: 001b3658 ib_mthca 0000:06:00.0: buf[07]: 00000000 ib_mthca 0000:06:00.0: buf[08]: 00000000 ib_mthca 0000:06:00.0: buf[09]: 00000000 ib_mthca 0000:06:00.0: buf[0a]: 00000000 ib_mthca 0000:06:00.0: buf[0b]: 00000000 ib_mthca 0000:06:00.0: buf[0c]: 00000000 ib_mthca 0000:06:00.0: buf[0d]: 00000000 ib_mthca 0000:06:00.0: buf[0e]: 00000000 ib_mthca 0000:06:00.0: buf[0f]: 00000000 ACPI: PCI interrupt for device 0000:06:00.0 disabled ib_mthca: Initializing 0000:06:00.0 ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 56 (level, low) -> IRQ 52 user_mad: process infiniband did not enable P_Key index support. user_mad: Documentation/infiniband/user_mad.txt has info on the new ABI. #> #> uname -a Linux (none) 2.6.24.02.02.08 #22 SMP Thu Feb 26 13:39:02 PST 2009 ia64 unknown #> lspci -v -d 15b3: 0000:04:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20) Subsystem: Hewlett-Packard Company: Unknown device 170a Flags: bus master, fast devsel, latency 0, IRQ 50 Memory at 00000000b0100000 (64-bit, non-prefetchable) [size=1M] Memory at 0000080380000000 (64-bit, prefetchable) [size=8M] Expansion ROM at 00000000b0000000 [disabled] [size=1M] Capabilities: [40] Power Management version 2 Capabilities: [48] Vital Product Data Capabilities: [90] Message Signalled Interrupts: 64bit+ Queue=0/5 Enable- Capabilities: [84] #11 [801f] Capabilities: [60] #10 [0001] 0000:06:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20) Subsystem: Hewlett-Packard Company: Unknown device 170a Flags: bus master, fast devsel, latency 0, IRQ 52 Memory at 00000000c0100000 (64-bit, non-prefetchable) [size=1M] Memory at 0000080480000000 (64-bit, prefetchable) [size=8M] Expansion ROM at 00000000c0000000 [disabled] [size=1M] Capabilities: [40] Power Management version 2 Capabilities: [48] Vital Product Data Capabilities: [90] Message Signalled Interrupts: 64bit+ Queue=0/5 Enable- Capabilities: [84] #11 [801f] Capabilities: [60] #10 [0001] 0000:08:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20) Subsystem: Hewlett-Packard Company: Unknown device 170a Flags: bus master, fast devsel, latency 0, IRQ 54 Memory at 00000000f0100000 (64-bit, non-prefetchable) [size=1M] Memory at 0000080780000000 (64-bit, prefetchable) [size=8M] Expansion ROM at 00000000f0000000 [disabled] [size=1M] Capabilities: [40] Power Management version 2 Capabilities: [48] Vital Product Data Capabilities: [90] Message Signalled Interrupts: 64bit+ Queue=0/5 Enable- Capabilities: [84] #11 [801f] Capabilities: [60] #10 [0001] #> cat /proc/cpuinfo processor : 0 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 0 core id : 0 thread id : 0 processor : 1 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 0 core id : 0 thread id : 1 processor : 2 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 0 core id : 1 thread id : 0 processor : 3 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 0 core id : 1 thread id : 1 processor : 4 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 1 core id : 0 thread id : 0 processor : 5 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 1 core id : 0 thread id : 1 processor : 6 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 1 core id : 1 thread id : 0 processor : 7 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 1 core id : 1 thread id : 1 processor : 8 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 2 core id : 0 thread id : 0 processor : 9 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 2 core id : 0 thread id : 1 processor : 10 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 2 core id : 1 thread id : 0 processor : 11 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 2 core id : 1 thread id : 1 processor : 12 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 3 core id : 0 thread id : 0 processor : 13 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 3 core id : 0 thread id : 1 processor : 14 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 3 core id : 1 thread id : 0 processor : 15 vendor : GenuineIntel arch : IA-64 family : 32 model : 1 model name : Dual-Core Intel(R) Itanium(R) Processor 9140N revision : 1 archrev : 0 features : branchlong, 16-byte atomic ops cpu number : 0 cpu regs : 4 cpu MHz : 1594.670 itc MHz : 399.165948 BogoMIPS : 3186.68 siblings : 4 physical id: 3 core id : 1 thread id : 1 #> cat /proc/meminfo MemTotal: 37277888 kB MemFree: 37036048 kB Buffers: 98304 kB Cached: 10320 kB SwapCached: 0 kB Active: 8128 kB Inactive: 101568 kB SwapTotal: 0 kB SwapFree: 0 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 1248 kB Mapped: 1792 kB Slab: 28400 kB SReclaimable: 1248 kB SUnreclaim: 27152 kB PageTables: 320 kB NFS_Unstable: 0 kB Bounce: 0 kB CommitLimit: 18638944 kB Committed_AS: 0 kB VmallocTotal: 137426880512 kB VmallocUsed: 432 kB VmallocChunk: 137426879584 kB -------------- next part -------------- An HTML attachment was scrubbed... URL: From keshetti.mahesh at gmail.com Wed Mar 11 23:22:11 2009 From: keshetti.mahesh at gmail.com (Keshetti Mahesh) Date: Thu, 12 Mar 2009 11:52:11 +0530 Subject: [ofa-general] ibdmchk: How to generate PSL and SL2VL files ? Message-ID: <829ded920903112322r728b65d8h3a78b67141303b3b@mail.gmail.com> Hi, Current implementation of 'ibdmchk' in cluster verification mode requires path<->SL(PSL) and SL2VL files as input. And the help(-h) message says that these files can be generated by OpenSM with '-D 0x43'. But I am not able to generate any such file with OpenSM. And there isn't any information related to these files in man page as well as help(-h). Can anyone please tell me how to generate path<->SL and SL2VL files using OpenSM? -Mahesh From keshetti.mahesh at gmail.com Wed Mar 11 23:26:51 2009 From: keshetti.mahesh at gmail.com (Keshetti Mahesh) Date: Thu, 12 Mar 2009 11:56:51 +0530 Subject: [ofa-general] Re: ibdmchk: How to generate PSL and SL2VL files ? In-Reply-To: <829ded920903112322r728b65d8h3a78b67141303b3b@mail.gmail.com> References: <829ded920903112322r728b65d8h3a78b67141303b3b@mail.gmail.com> Message-ID: <829ded920903112326s34486709s67538f900d6e5943@mail.gmail.com> On Thu, Mar 12, 2009 at 11:52 AM, Keshetti Mahesh wrote: > Hi, > > Current implementation of 'ibdmchk' in cluster verification mode > requires path<->SL(PSL) and SL2VL files as input. And the help(-h) > message says that these files can be generated by OpenSM with '-D 0x43'. > > But I am not able to generate any such file with OpenSM. And there isn't any > information related to these files in man page as well as help(-h). Can anyone > please tell me how to generate path<->SL and SL2VL files using OpenSM? > I am using OpenSM of OFED-1.4 distribution. -Mahesh From ogerlitz at voltaire.com Thu Mar 12 00:10:20 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 12 Mar 2009 09:10:20 +0200 Subject: [ofa-general] RE: tcp/rdma port unification patch in librdmacm In-Reply-To: <49B7D41C.4030707@opengridcomputing.com> References: <49B17E15.4050406@opengridcomputing.com> <811553D6D34A4E6F9AB686CA2E441D35@amr.corp.intel.com> <49B61C5D.6060900@Voltaire.com> <3281521B4F4A415A846A9F2E8AF9C21F@amr.corp.intel.com> <49B767DB.2050700@voltaire.com> <49B7D41C.4030707@opengridcomputing.com> Message-ID: <49B8B55C.50105@voltaire.com> Steve Wise wrote: > Or maybe map PS_TCP port space to the be the unified space, and have a > new PS_RDMA or PS_IB port space that doesn't? At and second Please note that the rdma-cm derives the IB Service-ID from the port space and the port provided to rdma_resolve_addr - and this is done along the lines of the IBTA "RDMA IP CM Service" annex A11. So we would need to have PS_IB_UD and PS_IB_RC that SID(ps=TCP, port=X) = SID(ps=IB_RC, port=X) at first thought this is possible and would add too much compatibility issues - except for the need to provide an app with the means to sense if the underlying librdmacm support the _IB port spaces. Still, I would like to avoid the case where someone runs MPI job over RDMA-CM over IB and librdmacm opens/bounds to all these sockets - so the preferred solution - for my taste would be some setting done by the user to have librdmacm allocate socket / carve a port - and it wouldn't be the default. For example, using an environment variable, e.g similar to the libibverbs fork support. Or. Or. From ogerlitz at Voltaire.com Thu Mar 12 00:51:33 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Thu, 12 Mar 2009 09:51:33 +0200 Subject: [ofa-general] Re: tcp/rdma port unification - iscsi/ddp/toe vs iwarp/ddp/toe In-Reply-To: <49B7D899.1030601@opengridcomputing.com> References: <49B17E15.4050406@opengridcomputing.com> <49B76CD5.705@voltaire.com> <49B7D3A0.5060307@opengridcomputing.com> <15ddcffd0903110819s3b83cec0vb87ee6114496858e@mail.gmail.com> <49B7D899.1030601@opengridcomputing.com> Message-ID: <49B8BF05.5030004@Voltaire.com> Steve Wise wrote: > Depends on the 4-tuple. The demux done in the TOE hw uses src addr/port and dst addr/port. Hi Steve, Trying to better understand the problem and possible solutions, I looked on the Chelsio iscsi offload driver (drivers/scsi/cxgb3i/* & Documentation/scsi/cxgb3i.txt) - which basically should suffer from the same issue - how to distinguish between a iscsi/os-tcp connection to iscsi/hw-tcp (toe) connection (or to iperf connection, etc as we discussed). I saw it manages the source port from its own data-structure (similar to the rdma-cm) but it doesn't open of a socket / nor bind the port - so I assume that a host opens two iscsi/tcp connections to a target on the same src ip/port - a party starts... At this point I came a cross the "cxgb3: manage private iSCSI IP address" patch whose change log states that its mostly ARP related- is the ARP resolution for iSCSI/DDP/TOE different from iWARP/DDP/TOE? does the below patch/approach eliminates the port space problem for iscsi/ddp? Or. commit a109a5b916bc180e14fad0d1e9c37a08c85652c0 Author: Karen Xie Date: Thu Dec 18 22:56:20 2008 -0800 cxgb3: manage private iSCSI IP address The accelerated iSCSI traffic could use a private IP address unknown to the OS: - The IP address is required in both drivers to manage ARP requests and connection set up. - Added an control call to retrieve the ip address. - Reply to ARP requests dedicated to the private IP address. Signed-off-by: Divy Le Ray Signed-off-by: Karen Xie Signed-off-by: David S. Miller From ogerlitz at voltaire.com Thu Mar 12 02:18:20 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 12 Mar 2009 11:18:20 +0200 (IST) Subject: [ofa-general] can't use perfquery from git against late kernels Message-ID: Sasha, I just case across a problem where with the latest management git - I can't use perfquery against latest kernel (I also see the problem if running against the drivers provided by ofed 1.4.x, but not against 1.3.y !) The problem doesn't exist for smpquery - and also it doesn't exist if I use perfquery from 1.4.x / 1.3.y - so it seems something was broken for user space GMPs between late kernels and latest mng git. --> run perfquery of 1.4.x # rpm -qf /usr/sbin/perfquery infiniband-diags-1.4.3_20081127_a92a5e5-1.ofed1.4.rc6 # /usr/sbin/perfquery 4 1 # Port counters: Lid 4 port 1 PortSelect:......................1 [... and so on all counters - works fine ...] --> run perfquery built from latest git # /home/ogerlitz/ib-mng/sbin/perfquery 4 1 ibwarn: [8278] mad_rpc: _do_madrpc failed; dport (Lid 4) /home/ogerlitz/ib-mng/sbin/perfquery: iberror: failed: classportinfo query running with -ddd (attached the two runs) - it seems that the there's no response on the classportinfo PM mad. Or. -------------- next part -------------- send buf 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 8001 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0104 0101 0000 0000 2d21 6022 24ff 756e 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 rcv buf 0104 0101 0000 0000 0000 005c 24ff 756e 0001 0000 0000 0000 0100 0000 0000 0000 001f a004 ff7f 0000 0f25 5dfc f17f 0000 1025 5dfc f17f 0000 0000 0000 0000 0000 c71d a004 ff7f 0000 5e26 44d3 3e00 0000 203c 3cfc f17f 0000 7834 5f30 2f70 6f72 c01e a004 ff7f 0000 7973 2f64 0000 0000 e019 a004 ff7f 0000 d01b a004 ff7f 0000 101b a004 ff7f 0000 60b9 74d3 3e00 0000 101f a004 ff7f 0000 f024 5dfc f17f 0000 60a7 74d3 3e00 0000 0000 0000 0100 0000 593f 3cfc f17f 0000 0800 0000 0000 0000 0000 0000 0000 0000 ffff ffff ffff ffff 1800 0000 2b00 0000 f924 5dfc f17f 0000 0000 0000 0000 0000 c81d a004 ff7f 0000 e644 3cfc 0300 0000 0000 0000 0000 0000 rcv buf 0104 0101 0000 0000 0000 005c 24ff 756e 0001 0000 0000 0000 0100 0000 0000 0000 001f a004 ff7f 0000 0f25 5dfc f17f 0000 1025 5dfc f17f 0000 0000 0000 0000 0000 c71d a004 ff7f 0000 5e26 44d3 3e00 0000 203c 3cfc f17f 0000 7834 5f30 2f70 6f72 c01e a004 ff7f 0000 7973 2f64 0000 0000 e019 a004 ff7f 0000 d01b a004 ff7f 0000 101b a004 ff7f 0000 60b9 74d3 3e00 0000 101f a004 ff7f 0000 f024 5dfc f17f 0000 60a7 74d3 3e00 0000 0000 0000 0100 0000 593f 3cfc f17f 0000 0800 0000 0000 0000 0000 0000 0000 0000 ffff ffff ffff ffff 1800 0000 2b00 0000 f924 5dfc f17f 0000 0000 0000 0000 0000 c81d a004 ff7f 0000 e644 3cfc 0300 0000 0000 0000 0000 0000 rcv buf 0104 0101 0000 0000 0000 005c 24ff 756e 0001 0000 0000 0000 0100 0000 0000 0000 001f a004 ff7f 0000 0f25 5dfc f17f 0000 1025 5dfc f17f 0000 0000 0000 0000 0000 c71d a004 ff7f 0000 5e26 44d3 3e00 0000 203c 3cfc f17f 0000 7834 5f30 2f70 6f72 c01e a004 ff7f 0000 7973 2f64 0000 0000 e019 a004 ff7f 0000 d01b a004 ff7f 0000 101b a004 ff7f 0000 60b9 74d3 3e00 0000 101f a004 ff7f 0000 f024 5dfc f17f 0000 60a7 74d3 3e00 0000 0000 0000 0100 0000 593f 3cfc f17f 0000 0800 0000 0000 0000 0000 0000 0000 0000 ffff ffff ffff ffff 1800 0000 2b00 0000 f924 5dfc f17f 0000 0000 0000 0000 0000 c81d a004 ff7f 0000 e644 3cfc 0300 0000 0000 0000 0000 0000 ibwarn: [8304] umad_init: umad_init ibwarn: [8304] umad_open_port: ca (null) port 0 ibwarn: [8304] umad_get_cas_names: max 20 ibwarn: [8304] umad_get_cas_names: return 1 cas ibwarn: [8304] resolve_ca_name: checking ca 'mlx4_0' ibwarn: [8304] resolve_ca_port: checking ca 'mlx4_0' ibwarn: [8304] umad_get_ca: ca_name mlx4_0 ibwarn: [8304] umad_get_ca: opened mlx4_0 ibwarn: [8304] resolve_ca_port: checking port 0 ibwarn: [8304] resolve_ca_port: checking port 1 ibwarn: [8304] resolve_ca_port: found active port 1 ibwarn: [8304] resolve_ca_name: found ca mlx4_0 with port 1 type 1 ibwarn: [8304] resolve_ca_name: found ca mlx4_0 with active port 1 ibwarn: [8304] umad_open_port: opening mlx4_0 port 1 ibwarn: [8304] dev_to_umad_id: mapped mlx4_0 1 to 0 ibwarn: [8304] umad_open_port: opened /dev/infiniband/umad0 fd 3 portid 0 ibwarn: [8304] umad_register: fd 3 mgmt_class 1 mgmt_version 1 rmpp_version 0 method_mask (nil) ibwarn: [8304] umad_register: fd 3 registered to use agent 0 qp 0 ibwarn: [8304] umad_register: fd 3 mgmt_class 129 mgmt_version 1 rmpp_version 0 method_mask (nil) ibwarn: [8304] umad_register: fd 3 registered to use agent 1 qp 0 ibwarn: [8304] umad_register: fd 3 mgmt_class 3 mgmt_version 2 rmpp_version 1 method_mask (nil) ibwarn: [8304] umad_register: fd 3 registered to use agent 2 qp 1 ibwarn: [8304] umad_register: fd 3 mgmt_class 4 mgmt_version 1 rmpp_version 0 method_mask (nil) ibwarn: [8304] umad_register: fd 3 registered to use agent 3 qp 1 ibwarn: [8304] pma_query_via: lid 4 port 1 ibwarn: [8304] umad_set_addr: umad 0x7fff04a01bc0 dlid 4 dqp 1 sl 0, qkey 80010000 ibwarn: [8304] _do_madrpc: >>> sending: len 256 pktsz 320 ibwarn: [8304] umad_send: fd 3 agentid 0 umad 0x7fff04a01bc0 timeout 1000 ibwarn: [8304] umad_dump: agent id 0 status 0 timeout 1000 ibwarn: [8304] umad_addr_dump: qpn 1 qkey 0x80010000 lid 4 sl 0 grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x0 Gid 0x00000000000000000000000000000000 ibwarn: [8304] umad_recv: fd 3 umad 0x7fff04a017c0 timeout 1000 ibwarn: [8304] umad_recv: mad received by agent 0 length 88 ibwarn: [8304] _do_madrpc: rcv buf: ibwarn: [8304] _do_madrpc: retry 1 (timeout 1000 ms) ibwarn: [8304] umad_send: fd 3 agentid 0 umad 0x7fff04a01bc0 timeout 1000 ibwarn: [8304] umad_dump: agent id 0 status 0 timeout 1000 ibwarn: [8304] umad_addr_dump: qpn 1 qkey 0x80010000 lid 4 sl 0 grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x0 Gid 0x00000000000000000000000000000000 ibwarn: [8304] umad_recv: fd 3 umad 0x7fff04a017c0 timeout 1000 ibwarn: [8304] umad_recv: mad received by agent 0 length 88 ibwarn: [8304] _do_madrpc: rcv buf: ibwarn: [8304] _do_madrpc: retry 2 (timeout 1000 ms) ibwarn: [8304] umad_send: fd 3 agentid 0 umad 0x7fff04a01bc0 timeout 1000 ibwarn: [8304] umad_dump: agent id 0 status 0 timeout 1000 ibwarn: [8304] umad_addr_dump: qpn 1 qkey 0x80010000 lid 4 sl 0 grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x0 Gid 0x00000000000000000000000000000000 ibwarn: [8304] umad_recv: fd 3 umad 0x7fff04a017c0 timeout 1000 ibwarn: [8304] umad_recv: mad received by agent 0 length 88 ibwarn: [8304] _do_madrpc: rcv buf: ibwarn: [8304] _do_madrpc: timeout after 3 retries, 3000 ms ibwarn: [8304] mad_rpc: _do_madrpc failed; dport (Lid 4) /home/ogerlitz/ib-mng/sbin/perfquery: iberror: [pid 8304] main: failed: classportinfo query -------------- next part -------------- send buf 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 8001 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0104 0101 0000 0000 4866 a330 42b8 9877 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 rcv buf 0104 0181 0000 0000 0000 004b 42b8 9877 0001 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0101 1000 0000 0014 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 mad data 0101 1000 0000 0014 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 send buf 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0001 8001 0000 0004 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0104 0101 0000 0000 4866 a330 42b8 9878 0012 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0101 1000 0000 0014 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 rcv buf 0104 0181 0000 0000 0000 004b 42b8 9878 0012 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0101 1000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0006 1e23 0005 048b 0000 15c1 0000 11d0 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 mad data 0101 1000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0006 1e23 0005 048b 0000 15c1 0000 11d0 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 ibwarn: [8273] umad_init: umad_init ibwarn: [8273] umad_open_port: ca (null) port 0 ibwarn: [8273] umad_get_cas_names: max 20 ibwarn: [8273] umad_get_cas_names: return 1 cas ibwarn: [8273] resolve_ca_name: checking ca 'mlx4_0' ibwarn: [8273] resolve_ca_port: checking ca 'mlx4_0' ibwarn: [8273] umad_get_ca: ca_name mlx4_0 ibwarn: [8273] umad_get_ca: opened mlx4_0 ibwarn: [8273] resolve_ca_port: checking port 0 ibwarn: [8273] resolve_ca_port: checking port 1 ibwarn: [8273] resolve_ca_port: found active port 1 ibwarn: [8273] resolve_ca_name: found ca mlx4_0 with port 1 type 1 ibwarn: [8273] resolve_ca_name: found ca mlx4_0 with active port 1 ibwarn: [8273] umad_open_port: opening mlx4_0 port 1 ibwarn: [8273] dev_to_umad_id: mapped mlx4_0 1 to 0 ibwarn: [8273] umad_open_port: opened /dev/infiniband/umad0 fd 3 portid 0 ibwarn: [8273] umad_register: fd 3 mgmt_class 1 mgmt_version 1 rmpp_version 0 method_mask (nil) ibwarn: [8273] umad_register: fd 3 registered to use agent 0 qp 0 ibwarn: [8273] umad_register: fd 3 mgmt_class 129 mgmt_version 1 rmpp_version 0 method_mask (nil) ibwarn: [8273] umad_register: fd 3 registered to use agent 1 qp 0 ibwarn: [8273] umad_register: fd 3 mgmt_class 3 mgmt_version 2 rmpp_version 1 method_mask (nil) ibwarn: [8273] umad_register: fd 3 registered to use agent 2 qp 1 ibwarn: [8273] umad_register: fd 3 mgmt_class 4 mgmt_version 1 rmpp_version 0 method_mask (nil) ibwarn: [8273] umad_register: fd 3 registered to use agent 3 qp 1 ibwarn: [8273] pma_query_via: lid 4 port 1 ibwarn: [8273] umad_set_addr: umad 0x7fffe8c9a620 dlid 4 dqp 1 sl 0, qkey 80010000 ibwarn: [8273] _do_madrpc: >>> sending: len 256 pktsz 320 ibwarn: [8273] umad_send: fd 3 agentid 3 umad 0x7fffe8c9a620 timeout 1000 ibwarn: [8273] umad_dump: agent id 3 status 0 timeout 1000 ibwarn: [8273] umad_addr_dump: qpn 1 qkey 0x80010000 lid 0x4 sl 0 grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x0 Gid 0x00000000000000000000000000000000 ibwarn: [8273] umad_recv: fd 3 umad 0x7fffe8c9a220 timeout 1000 ibwarn: [8273] umad_recv: mad received by agent 3 length 320 ibwarn: [8273] _do_madrpc: rcv buf: ibwarn: [8273] mad_rpc: data offs 64 sz 192 ibwarn: [8273] pma_query_via: lid 4 port 1 ibwarn: [8273] umad_set_addr: umad 0x7fffe8c9a1d0 dlid 4 dqp 1 sl 0, qkey 80010000 ibwarn: [8273] _do_madrpc: >>> sending: len 256 pktsz 320 ibwarn: [8273] umad_send: fd 3 agentid 3 umad 0x7fffe8c9a1d0 timeout 1000 ibwarn: [8273] umad_dump: agent id 3 status 0 timeout 1000 ibwarn: [8273] umad_addr_dump: qpn 1 qkey 0x80010000 lid 0x4 sl 0 grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x0 Gid 0x00000000000000000000000000000000 ibwarn: [8273] umad_recv: fd 3 umad 0x7fffe8c99dd0 timeout 1000 ibwarn: [8273] umad_recv: mad received by agent 3 length 320 ibwarn: [8273] _do_madrpc: rcv buf: ibwarn: [8273] mad_rpc: data offs 64 sz 192 # Port counters: Lid 4 port 1 PortSelect:......................1 CounterSelect:...................0x1000 SymbolErrors:....................0 LinkRecovers:....................0 LinkDowned:......................0 RcvErrors:.......................0 RcvRemotePhysErrors:.............0 RcvSwRelayErrors:................0 XmtDiscards:.....................0 XmtConstraintErrors:.............0 RcvConstraintErrors:.............0 LinkIntegrityErrors:.............0 ExcBufOverrunErrors:.............0 VL15Dropped:.....................0 XmtData:.........................400931 RcvData:.........................328843 XmtPkts:.........................5569 RcvPkts:.........................4560 From kliteyn at dev.mellanox.co.il Thu Mar 12 02:30:18 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Thu, 12 Mar 2009 11:30:18 +0200 Subject: [ofa-general] [PATCH] opensm/osm_ucast_ftree.c: remove __osm_ftree prefix in static functions Message-ID: <49B8D62A.6060107@dev.mellanox.co.il> Hi Sasha, Removing all the __osm_ftree prefixes in static functions naming. Signed-off-by: Yevgeny Kliteynik --- opensm/opensm/osm_ucast_ftree.c | 1074 +++++++++++++++++++-------------------- 1 files changed, 524 insertions(+), 550 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index 6e2e7de..4e52d4b 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -217,8 +217,8 @@ typedef struct ftree_fabric_t_ { ** ***************************************************/ -static int OSM_CDECL __osm_ftree_compare_switches_by_index(IN const void *p1, - IN const void *p2) +static int OSM_CDECL compare_switches_by_index(IN const void *p1, + IN const void *p2) { ftree_sw_t **pp_sw1 = (ftree_sw_t **) p1; ftree_sw_t **pp_sw2 = (ftree_sw_t **) p2; @@ -236,18 +236,15 @@ static int OSM_CDECL __osm_ftree_compare_switches_by_index(IN const void *p1, /***************************************************/ static int OSM_CDECL -__osm_ftree_compare_port_groups_by_remote_switch_index(IN const void *p1, - IN const void *p2) +compare_port_groups_by_remote_switch_index(IN const void *p1, + IN const void *p2) { ftree_port_group_t **pp_g1 = (ftree_port_group_t **) p1; ftree_port_group_t **pp_g2 = (ftree_port_group_t **) p2; return - __osm_ftree_compare_switches_by_index(& - ((*pp_g1)->remote_hca_or_sw. - p_sw), - &((*pp_g2)->remote_hca_or_sw. - p_sw)); + compare_switches_by_index(&((*pp_g1)->remote_hca_or_sw.p_sw), + &((*pp_g2)->remote_hca_or_sw.p_sw)); } /*************************************************** @@ -256,14 +253,14 @@ __osm_ftree_compare_port_groups_by_remote_switch_index(IN const void *p1, ** ***************************************************/ -static void __osm_ftree_tuple_init(IN ftree_tuple_t tuple) +static void tuple_init(IN ftree_tuple_t tuple) { memset(tuple, 0xFF, FTREE_TUPLE_LEN); } /***************************************************/ -static inline boolean_t __osm_ftree_tuple_assigned(IN ftree_tuple_t tuple) +static inline boolean_t tuple_assigned(IN ftree_tuple_t tuple) { return (tuple[0] != 0xFF); } @@ -272,14 +269,14 @@ static inline boolean_t __osm_ftree_tuple_assigned(IN ftree_tuple_t tuple) #define FTREE_TUPLE_BUFFERS_NUM 6 -static char *__osm_ftree_tuple_to_str(IN ftree_tuple_t tuple) +static char *tuple_to_str(IN ftree_tuple_t tuple) { static char buffer[FTREE_TUPLE_BUFFERS_NUM][FTREE_TUPLE_BUFF_LEN]; static uint8_t ind = 0; char *ret_buffer; uint32_t i; - if (!__osm_ftree_tuple_assigned(tuple)) + if (!tuple_assigned(tuple)) return "INDEX.NOT.ASSIGNED"; buffer[ind][0] = '\0'; @@ -295,11 +292,11 @@ static char *__osm_ftree_tuple_to_str(IN ftree_tuple_t tuple) ret_buffer = buffer[ind]; ind = (ind + 1) % FTREE_TUPLE_BUFFERS_NUM; return ret_buffer; -} /* __osm_ftree_tuple_to_str() */ +} /* tuple_to_str() */ /***************************************************/ -static inline ftree_tuple_key_t __osm_ftree_tuple_to_key(IN ftree_tuple_t tuple) +static inline ftree_tuple_key_t tuple_to_key(IN ftree_tuple_t tuple) { ftree_tuple_key_t key; memcpy(&key, tuple, FTREE_TUPLE_LEN); @@ -308,7 +305,7 @@ static inline ftree_tuple_key_t __osm_ftree_tuple_to_key(IN ftree_tuple_t tuple) /***************************************************/ -static inline void __osm_ftree_tuple_from_key(IN ftree_tuple_t tuple, +static inline void tuple_from_key(IN ftree_tuple_t tuple, IN ftree_tuple_key_t key) { memcpy(tuple, &key, FTREE_TUPLE_LEN); @@ -320,8 +317,7 @@ static inline void __osm_ftree_tuple_from_key(IN ftree_tuple_t tuple, ** ***************************************************/ -static ftree_sw_tbl_element_t *__osm_ftree_sw_tbl_element_create(IN ftree_sw_t * - p_sw) +static ftree_sw_tbl_element_t *sw_tbl_element_create(IN ftree_sw_t * p_sw) { ftree_sw_tbl_element_t *p_element = (ftree_sw_tbl_element_t *) malloc(sizeof(ftree_sw_tbl_element_t)); @@ -335,8 +331,7 @@ static ftree_sw_tbl_element_t *__osm_ftree_sw_tbl_element_create(IN ftree_sw_t * /***************************************************/ -static void __osm_ftree_sw_tbl_element_destroy(IN ftree_sw_tbl_element_t * - p_element) +static void sw_tbl_element_destroy(IN ftree_sw_tbl_element_t * p_element) { if (!p_element) return; @@ -349,8 +344,8 @@ static void __osm_ftree_sw_tbl_element_destroy(IN ftree_sw_tbl_element_t * ** ***************************************************/ -static ftree_port_t *__osm_ftree_port_create(IN uint8_t port_num, - IN uint8_t remote_port_num) +static ftree_port_t *port_create(IN uint8_t port_num, + IN uint8_t remote_port_num) { ftree_port_t *p_port = (ftree_port_t *) malloc(sizeof(ftree_port_t)); if (!p_port) @@ -365,7 +360,7 @@ static ftree_port_t *__osm_ftree_port_create(IN uint8_t port_num, /***************************************************/ -static void __osm_ftree_port_destroy(IN ftree_port_t * p_port) +static void port_destroy(IN ftree_port_t * p_port) { if (p_port) free(p_port); @@ -378,18 +373,18 @@ static void __osm_ftree_port_destroy(IN ftree_port_t * p_port) ***************************************************/ static ftree_port_group_t * -__osm_ftree_port_group_create(IN ib_net16_t base_lid, - IN ib_net16_t remote_base_lid, - IN ib_net64_t port_guid, - IN ib_net64_t node_guid, - IN uint8_t node_type, - IN void *p_hca_or_sw, - IN ib_net64_t remote_port_guid, - IN ib_net64_t remote_node_guid, - IN uint8_t remote_node_type, - IN void *p_remote_hca_or_sw, - IN boolean_t is_cn, - IN boolean_t is_io) +port_group_create(IN ib_net16_t base_lid, + IN ib_net16_t remote_base_lid, + IN ib_net64_t port_guid, + IN ib_net64_t node_guid, + IN uint8_t node_type, + IN void *p_hca_or_sw, + IN ib_net64_t remote_port_guid, + IN ib_net64_t remote_node_guid, + IN uint8_t remote_node_type, + IN void *p_remote_hca_or_sw, + IN boolean_t is_cn, + IN boolean_t is_io) { ftree_port_group_t *p_group = (ftree_port_group_t *) malloc(sizeof(ftree_port_group_t)); @@ -439,11 +434,11 @@ __osm_ftree_port_group_create(IN ib_net16_t base_lid, p_group->is_cn = is_cn; p_group->is_io = is_io; return p_group; -} /* __osm_ftree_port_group_create() */ +} /* port_group_create() */ /***************************************************/ -static void __osm_ftree_port_group_destroy(IN ftree_port_group_t * p_group) +static void port_group_destroy(IN ftree_port_group_t * p_group) { uint32_t i; uint32_t size; @@ -456,18 +451,18 @@ static void __osm_ftree_port_group_destroy(IN ftree_port_group_t * p_group) size = cl_ptr_vector_get_size(&p_group->ports); for (i = 0; i < size; i++) { cl_ptr_vector_at(&p_group->ports, i, (void *)&p_port); - __osm_ftree_port_destroy(p_port); + port_destroy(p_port); } cl_ptr_vector_destroy(&p_group->ports); free(p_group); -} /* __osm_ftree_port_group_destroy() */ +} /* port_group_destroy() */ /***************************************************/ static void -__osm_ftree_port_group_dump(IN ftree_fabric_t * p_ftree, - IN ftree_port_group_t * p_group, - IN ftree_direction_t direction) +port_group_dump(IN ftree_fabric_t * p_ftree, + IN ftree_port_group_t * p_group, + IN ftree_direction_t direction) { ftree_port_t *p_port; uint32_t size; @@ -504,13 +499,13 @@ __osm_ftree_port_group_dump(IN ftree_fabric_t * p_ftree, cl_ntoh64(p_group->remote_port_guid), cl_ntoh16(p_group->remote_base_lid)); -} /* __osm_ftree_port_group_dump() */ +} /* port_group_dump() */ /***************************************************/ static void -__osm_ftree_port_group_add_port(IN ftree_port_group_t * p_group, - IN uint8_t port_num, IN uint8_t remote_port_num) +port_group_add_port(IN ftree_port_group_t * p_group, + IN uint8_t port_num, IN uint8_t remote_port_num) { uint16_t i; ftree_port_t *p_port; @@ -521,7 +516,7 @@ __osm_ftree_port_group_add_port(IN ftree_port_group_t * p_group, return; } - p_port = __osm_ftree_port_create(port_num, remote_port_num); + p_port = port_create(port_num, remote_port_num); cl_ptr_vector_insert(&p_group->ports, p_port, NULL); } @@ -531,8 +526,8 @@ __osm_ftree_port_group_add_port(IN ftree_port_group_t * p_group, ** ***************************************************/ -static ftree_sw_t *__osm_ftree_sw_create(IN ftree_fabric_t * p_ftree, - IN osm_switch_t * p_osm_sw) +static ftree_sw_t *sw_create(IN ftree_fabric_t * p_ftree, + IN osm_switch_t * p_osm_sw) { ftree_sw_t *p_sw; uint8_t ports_num; @@ -548,7 +543,7 @@ static ftree_sw_t *__osm_ftree_sw_create(IN ftree_fabric_t * p_ftree, p_sw->p_osm_sw = p_osm_sw; p_sw->rank = 0xFFFFFFFF; - __osm_ftree_tuple_init(p_sw->tuple); + tuple_init(p_sw->tuple); p_sw->base_lid = osm_node_get_base_lid(p_sw->p_osm_sw->p_node, 0); @@ -566,12 +561,12 @@ static ftree_sw_t *__osm_ftree_sw_create(IN ftree_fabric_t * p_ftree, memset(p_osm_sw->new_lft, OSM_NO_PATH, IB_LID_UCAST_END_HO + 1); return p_sw; -} /* __osm_ftree_sw_create() */ +} /* sw_create() */ /***************************************************/ -static void __osm_ftree_sw_destroy(IN ftree_fabric_t * p_ftree, - IN ftree_sw_t * p_sw) +static void sw_destroy(IN ftree_fabric_t * p_ftree, + IN ftree_sw_t * p_sw) { uint8_t i; @@ -579,20 +574,20 @@ static void __osm_ftree_sw_destroy(IN ftree_fabric_t * p_ftree, return; for (i = 0; i < p_sw->down_port_groups_num; i++) - __osm_ftree_port_group_destroy(p_sw->down_port_groups[i]); + port_group_destroy(p_sw->down_port_groups[i]); for (i = 0; i < p_sw->up_port_groups_num; i++) - __osm_ftree_port_group_destroy(p_sw->up_port_groups[i]); + port_group_destroy(p_sw->up_port_groups[i]); if (p_sw->down_port_groups) free(p_sw->down_port_groups); if (p_sw->up_port_groups) free(p_sw->up_port_groups); free(p_sw); -} /* __osm_ftree_sw_destroy() */ +} /* sw_destroy() */ /***************************************************/ -static uint64_t __osm_ftree_sw_get_guid_no(IN ftree_sw_t * p_sw) +static uint64_t sw_get_guid_no(IN ftree_sw_t * p_sw) { if (!p_sw) return 0; @@ -601,15 +596,15 @@ static uint64_t __osm_ftree_sw_get_guid_no(IN ftree_sw_t * p_sw) /***************************************************/ -static uint64_t __osm_ftree_sw_get_guid_ho(IN ftree_sw_t * p_sw) +static uint64_t sw_get_guid_ho(IN ftree_sw_t * p_sw) { - return cl_ntoh64(__osm_ftree_sw_get_guid_no(p_sw)); + return cl_ntoh64(sw_get_guid_no(p_sw)); } /***************************************************/ -static void __osm_ftree_sw_dump(IN ftree_fabric_t * p_ftree, - IN ftree_sw_t * p_sw) +static void sw_dump(IN ftree_fabric_t * p_ftree, + IN ftree_sw_t * p_sw) { uint32_t i; @@ -622,23 +617,24 @@ static void __osm_ftree_sw_dump(IN ftree_fabric_t * p_ftree, OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Switch index: %s, GUID: 0x%016" PRIx64 ", Ports: %u DOWN, %u UP\n", - __osm_ftree_tuple_to_str(p_sw->tuple), - __osm_ftree_sw_get_guid_ho(p_sw), p_sw->down_port_groups_num, + tuple_to_str(p_sw->tuple), + sw_get_guid_ho(p_sw), p_sw->down_port_groups_num, p_sw->up_port_groups_num); for (i = 0; i < p_sw->down_port_groups_num; i++) - __osm_ftree_port_group_dump(p_ftree, - p_sw->down_port_groups[i], - FTREE_DIRECTION_DOWN); + port_group_dump(p_ftree, + p_sw->down_port_groups[i], + FTREE_DIRECTION_DOWN); for (i = 0; i < p_sw->up_port_groups_num; i++) - __osm_ftree_port_group_dump(p_ftree, p_sw->up_port_groups[i], - FTREE_DIRECTION_UP); + port_group_dump(p_ftree, + p_sw->up_port_groups[i], + FTREE_DIRECTION_UP); -} /* __osm_ftree_sw_dump() */ +} /* sw_dump() */ /***************************************************/ -static boolean_t __osm_ftree_sw_ranked(IN ftree_sw_t * p_sw) +static boolean_t sw_ranked(IN ftree_sw_t * p_sw) { return (p_sw->rank != 0xFFFFFFFF); } @@ -646,9 +642,9 @@ static boolean_t __osm_ftree_sw_ranked(IN ftree_sw_t * p_sw) /***************************************************/ static ftree_port_group_t * -__osm_ftree_sw_get_port_group_by_remote_lid(IN ftree_sw_t * p_sw, - IN ib_net16_t remote_base_lid, - IN ftree_direction_t direction) +sw_get_port_group_by_remote_lid(IN ftree_sw_t * p_sw, + IN ib_net16_t remote_base_lid, + IN ftree_direction_t direction) { uint32_t i; uint32_t size; @@ -667,39 +663,38 @@ __osm_ftree_sw_get_port_group_by_remote_lid(IN ftree_sw_t * p_sw, return port_groups[i]; return NULL; -} /* __osm_ftree_sw_get_port_group_by_remote_lid() */ +} /* sw_get_port_group_by_remote_lid() */ /***************************************************/ static void -__osm_ftree_sw_add_port(IN ftree_sw_t * p_sw, - IN uint8_t port_num, - IN uint8_t remote_port_num, - IN ib_net16_t base_lid, - IN ib_net16_t remote_base_lid, - IN ib_net64_t port_guid, - IN ib_net64_t remote_port_guid, - IN ib_net64_t remote_node_guid, - IN uint8_t remote_node_type, - IN void *p_remote_hca_or_sw, - IN ftree_direction_t direction) +sw_add_port(IN ftree_sw_t * p_sw, + IN uint8_t port_num, + IN uint8_t remote_port_num, + IN ib_net16_t base_lid, + IN ib_net16_t remote_base_lid, + IN ib_net64_t port_guid, + IN ib_net64_t remote_port_guid, + IN ib_net64_t remote_node_guid, + IN uint8_t remote_node_type, + IN void *p_remote_hca_or_sw, + IN ftree_direction_t direction) { ftree_port_group_t *p_group = - __osm_ftree_sw_get_port_group_by_remote_lid(p_sw, remote_base_lid, - direction); + sw_get_port_group_by_remote_lid(p_sw, remote_base_lid, direction); if (!p_group) { - p_group = __osm_ftree_port_group_create(base_lid, - remote_base_lid, - port_guid, - __osm_ftree_sw_get_guid_no - (p_sw), - IB_NODE_TYPE_SWITCH, - p_sw, remote_port_guid, - remote_node_guid, - remote_node_type, - p_remote_hca_or_sw, - FALSE, FALSE); + p_group = port_group_create(base_lid, + remote_base_lid, + port_guid, + sw_get_guid_no + (p_sw), + IB_NODE_TYPE_SWITCH, + p_sw, remote_port_guid, + remote_node_guid, + remote_node_type, + p_remote_hca_or_sw, + FALSE, FALSE); CL_ASSERT(p_group); if (direction == FTREE_DIRECTION_UP) @@ -709,16 +704,16 @@ __osm_ftree_sw_add_port(IN ftree_sw_t * p_sw, p_sw->down_port_groups[p_sw->down_port_groups_num++] = p_group; } - __osm_ftree_port_group_add_port(p_group, port_num, remote_port_num); + port_group_add_port(p_group, port_num, remote_port_num); -} /* __osm_ftree_sw_add_port() */ +} /* sw_add_port() */ /***************************************************/ static inline cl_status_t -__osm_ftree_sw_set_hops(IN ftree_sw_t * p_sw, - IN uint16_t lid_ho, IN uint8_t port_num, - IN uint8_t hops) +sw_set_hops(IN ftree_sw_t * p_sw, + IN uint16_t lid_ho, IN uint8_t port_num, + IN uint8_t hops) { /* set local min hop table(LID) */ return osm_switch_set_hops(p_sw->p_osm_sw, lid_ho, port_num, hops); @@ -730,7 +725,7 @@ __osm_ftree_sw_set_hops(IN ftree_sw_t * p_sw, ** ***************************************************/ -static ftree_hca_t *__osm_ftree_hca_create(IN osm_node_t * p_osm_node) +static ftree_hca_t *hca_create(IN osm_node_t * p_osm_node) { ftree_hca_t *p_hca = (ftree_hca_t *) malloc(sizeof(ftree_hca_t)); if (p_hca == NULL) @@ -749,7 +744,7 @@ static ftree_hca_t *__osm_ftree_hca_create(IN osm_node_t * p_osm_node) /***************************************************/ -static void __osm_ftree_hca_destroy(IN ftree_hca_t * p_hca) +static void hca_destroy(IN ftree_hca_t * p_hca) { uint32_t i; @@ -757,7 +752,7 @@ static void __osm_ftree_hca_destroy(IN ftree_hca_t * p_hca) return; for (i = 0; i < p_hca->up_port_groups_num; i++) - __osm_ftree_port_group_destroy(p_hca->up_port_groups[i]); + port_group_destroy(p_hca->up_port_groups[i]); if (p_hca->up_port_groups) free(p_hca->up_port_groups); @@ -767,7 +762,7 @@ static void __osm_ftree_hca_destroy(IN ftree_hca_t * p_hca) /***************************************************/ -static uint64_t __osm_ftree_hca_get_guid_no(IN ftree_hca_t * p_hca) +static uint64_t hca_get_guid_no(IN ftree_hca_t * p_hca) { if (!p_hca) return 0; @@ -776,14 +771,14 @@ static uint64_t __osm_ftree_hca_get_guid_no(IN ftree_hca_t * p_hca) /***************************************************/ -static uint64_t __osm_ftree_hca_get_guid_ho(IN ftree_hca_t * p_hca) +static uint64_t hca_get_guid_ho(IN ftree_hca_t * p_hca) { - return cl_ntoh64(__osm_ftree_hca_get_guid_no(p_hca)); + return cl_ntoh64(hca_get_guid_no(p_hca)); } /***************************************************/ -static void __osm_ftree_hca_dump(IN ftree_fabric_t * p_ftree, +static void hca_dump(IN ftree_fabric_t * p_ftree, IN ftree_hca_t * p_hca) { uint32_t i; @@ -796,18 +791,18 @@ static void __osm_ftree_hca_dump(IN ftree_fabric_t * p_ftree, OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "CA GUID: 0x%016" PRIx64 ", Ports: %u UP\n", - __osm_ftree_hca_get_guid_ho(p_hca), p_hca->up_port_groups_num); + hca_get_guid_ho(p_hca), p_hca->up_port_groups_num); for (i = 0; i < p_hca->up_port_groups_num; i++) - __osm_ftree_port_group_dump(p_ftree, p_hca->up_port_groups[i], - FTREE_DIRECTION_UP); + port_group_dump(p_ftree, p_hca->up_port_groups[i], + FTREE_DIRECTION_UP); } /***************************************************/ static ftree_port_group_t * -__osm_ftree_hca_get_port_group_by_remote_lid(IN ftree_hca_t * p_hca, - IN ib_net16_t remote_base_lid) +hca_get_port_group_by_remote_lid(IN ftree_hca_t * p_hca, + IN ib_net16_t remote_base_lid) { uint32_t i; for (i = 0; i < p_hca->up_port_groups_num; i++) @@ -821,17 +816,17 @@ __osm_ftree_hca_get_port_group_by_remote_lid(IN ftree_hca_t * p_hca, /***************************************************/ static void -__osm_ftree_hca_add_port(IN ftree_hca_t * p_hca, - IN uint8_t port_num, - IN uint8_t remote_port_num, - IN ib_net16_t base_lid, - IN ib_net16_t remote_base_lid, - IN ib_net64_t port_guid, - IN ib_net64_t remote_port_guid, - IN ib_net64_t remote_node_guid, - IN uint8_t remote_node_type, - IN void *p_remote_hca_or_sw, IN boolean_t is_cn, - IN boolean_t is_io) +hca_add_port(IN ftree_hca_t * p_hca, + IN uint8_t port_num, + IN uint8_t remote_port_num, + IN ib_net16_t base_lid, + IN ib_net16_t remote_base_lid, + IN ib_net64_t port_guid, + IN ib_net64_t remote_port_guid, + IN ib_net64_t remote_node_guid, + IN uint8_t remote_node_type, + IN void *p_remote_hca_or_sw, IN boolean_t is_cn, + IN boolean_t is_io) { ftree_port_group_t *p_group; @@ -839,27 +834,25 @@ __osm_ftree_hca_add_port(IN ftree_hca_t * p_hca, in hca's that lead to switches */ CL_ASSERT(remote_node_type == IB_NODE_TYPE_SWITCH); - p_group = - __osm_ftree_hca_get_port_group_by_remote_lid(p_hca, - remote_base_lid); + p_group = hca_get_port_group_by_remote_lid(p_hca, remote_base_lid); if (!p_group) { - p_group = __osm_ftree_port_group_create(base_lid, - remote_base_lid, - port_guid, - __osm_ftree_hca_get_guid_no - (p_hca), - IB_NODE_TYPE_CA, p_hca, - remote_port_guid, - remote_node_guid, - remote_node_type, - p_remote_hca_or_sw, - is_cn, is_io); + p_group = port_group_create(base_lid, + remote_base_lid, + port_guid, + hca_get_guid_no + (p_hca), + IB_NODE_TYPE_CA, p_hca, + remote_port_guid, + remote_node_guid, + remote_node_type, + p_remote_hca_or_sw, + is_cn, is_io); p_hca->up_port_groups[p_hca->up_port_groups_num++] = p_group; } - __osm_ftree_port_group_add_port(p_group, port_num, remote_port_num); + port_group_add_port(p_group, port_num, remote_port_num); -} /* __osm_ftree_hca_add_port() */ +} /* hca_add_port() */ /*************************************************** ** @@ -867,7 +860,7 @@ __osm_ftree_hca_add_port(IN ftree_hca_t * p_hca, ** ***************************************************/ -static ftree_fabric_t *__osm_ftree_fabric_create() +static ftree_fabric_t *fabric_create() { ftree_fabric_t *p_ftree = (ftree_fabric_t *) malloc(sizeof(ftree_fabric_t)); @@ -887,7 +880,7 @@ static ftree_fabric_t *__osm_ftree_fabric_create() /***************************************************/ -static void __osm_ftree_fabric_clear(ftree_fabric_t * p_ftree) +static void fabric_clear(ftree_fabric_t * p_ftree) { ftree_hca_t *p_hca; ftree_hca_t *p_next_hca; @@ -906,7 +899,7 @@ static void __osm_ftree_fabric_clear(ftree_fabric_t * p_ftree) while (p_next_hca != (ftree_hca_t *) cl_qmap_end(&p_ftree->hca_tbl)) { p_hca = p_next_hca; p_next_hca = (ftree_hca_t *) cl_qmap_next(&p_hca->map_item); - __osm_ftree_hca_destroy(p_hca); + hca_destroy(p_hca); } cl_qmap_remove_all(&p_ftree->hca_tbl); @@ -916,7 +909,7 @@ static void __osm_ftree_fabric_clear(ftree_fabric_t * p_ftree) while (p_next_sw != (ftree_sw_t *) cl_qmap_end(&p_ftree->sw_tbl)) { p_sw = p_next_sw; p_next_sw = (ftree_sw_t *) cl_qmap_next(&p_sw->map_item); - __osm_ftree_sw_destroy(p_ftree, p_sw); + sw_destroy(p_ftree, p_sw); } cl_qmap_remove_all(&p_ftree->sw_tbl); @@ -931,7 +924,7 @@ static void __osm_ftree_fabric_clear(ftree_fabric_t * p_ftree) p_next_element = (ftree_sw_tbl_element_t *) cl_qmap_next(&p_element-> map_item); - __osm_ftree_sw_tbl_element_destroy(p_element); + sw_tbl_element_destroy(p_element); } cl_qmap_remove_all(&p_ftree->sw_by_tuple_tbl); @@ -972,31 +965,31 @@ static void __osm_ftree_fabric_clear(ftree_fabric_t * p_ftree) p_ftree->leaf_switches = NULL; p_ftree->fabric_built = FALSE; -} /* __osm_ftree_fabric_destroy() */ +} /* fabric_destroy() */ /***************************************************/ -static void __osm_ftree_fabric_destroy(ftree_fabric_t * p_ftree) +static void fabric_destroy(ftree_fabric_t * p_ftree) { if (!p_ftree) return; - __osm_ftree_fabric_clear(p_ftree); + fabric_clear(p_ftree); free(p_ftree); } /***************************************************/ -static uint8_t __osm_ftree_fabric_get_rank(ftree_fabric_t * p_ftree) +static uint8_t fabric_get_rank(ftree_fabric_t * p_ftree) { return p_ftree->leaf_switch_rank + 1; } /***************************************************/ -static void __osm_ftree_fabric_add_hca(ftree_fabric_t * p_ftree, - osm_node_t * p_osm_node) +static void fabric_add_hca(ftree_fabric_t * p_ftree, + osm_node_t * p_osm_node) { - ftree_hca_t *p_hca = __osm_ftree_hca_create(p_osm_node); + ftree_hca_t *p_hca = hca_create(p_osm_node); CL_ASSERT(osm_node_get_type(p_osm_node) == IB_NODE_TYPE_CA); @@ -1006,10 +999,10 @@ static void __osm_ftree_fabric_add_hca(ftree_fabric_t * p_ftree, /***************************************************/ -static void __osm_ftree_fabric_add_sw(ftree_fabric_t * p_ftree, - osm_switch_t * p_osm_sw) +static void fabric_add_sw(ftree_fabric_t * p_ftree, + osm_switch_t * p_osm_sw) { - ftree_sw_t *p_sw = __osm_ftree_sw_create(p_ftree, p_osm_sw); + ftree_sw_t *p_sw = sw_create(p_ftree, p_osm_sw); CL_ASSERT(osm_node_get_type(p_osm_sw->p_node) == IB_NODE_TYPE_SWITCH); @@ -1023,32 +1016,30 @@ static void __osm_ftree_fabric_add_sw(ftree_fabric_t * p_ftree, /***************************************************/ -static void __osm_ftree_fabric_add_sw_by_tuple(IN ftree_fabric_t * p_ftree, - IN ftree_sw_t * p_sw) +static void fabric_add_sw_by_tuple(IN ftree_fabric_t * p_ftree, + IN ftree_sw_t * p_sw) { - CL_ASSERT(__osm_ftree_tuple_assigned(p_sw->tuple)); + CL_ASSERT(tuple_assigned(p_sw->tuple)); cl_qmap_insert(&p_ftree->sw_by_tuple_tbl, - __osm_ftree_tuple_to_key(p_sw->tuple), - &__osm_ftree_sw_tbl_element_create(p_sw)->map_item); + tuple_to_key(p_sw->tuple), + &sw_tbl_element_create(p_sw)->map_item); } /***************************************************/ -static ftree_sw_t *__osm_ftree_fabric_get_sw_by_tuple(IN ftree_fabric_t * - p_ftree, - IN ftree_tuple_t tuple) +static ftree_sw_t *fabric_get_sw_by_tuple(IN ftree_fabric_t * p_ftree, + IN ftree_tuple_t tuple) { ftree_sw_tbl_element_t *p_element; - CL_ASSERT(__osm_ftree_tuple_assigned(tuple)); + CL_ASSERT(tuple_assigned(tuple)); - __osm_ftree_tuple_to_key(tuple); + tuple_to_key(tuple); p_element = (ftree_sw_tbl_element_t *) cl_qmap_get(&p_ftree->sw_by_tuple_tbl, - __osm_ftree_tuple_to_key - (tuple)); + tuple_to_key(tuple)); if (p_element == (ftree_sw_tbl_element_t *) cl_qmap_end(&p_ftree->sw_by_tuple_tbl)) return NULL; @@ -1058,8 +1049,8 @@ static ftree_sw_t *__osm_ftree_fabric_get_sw_by_tuple(IN ftree_fabric_t * /***************************************************/ -static ftree_sw_t *__osm_ftree_fabric_get_sw_by_guid(IN ftree_fabric_t * - p_ftree, IN uint64_t guid) +static ftree_sw_t *fabric_get_sw_by_guid(IN ftree_fabric_t * p_ftree, + IN uint64_t guid) { ftree_sw_t *p_sw; p_sw = (ftree_sw_t *) cl_qmap_get(&p_ftree->sw_tbl, guid); @@ -1070,7 +1061,7 @@ static ftree_sw_t *__osm_ftree_fabric_get_sw_by_guid(IN ftree_fabric_t * /***************************************************/ -static ftree_hca_t *__osm_ftree_fabric_get_hca_by_guid(IN ftree_fabric_t * +static ftree_hca_t *fabric_get_hca_by_guid(IN ftree_fabric_t * p_ftree, IN uint64_t guid) { @@ -1083,7 +1074,7 @@ static ftree_hca_t *__osm_ftree_fabric_get_hca_by_guid(IN ftree_fabric_t * /***************************************************/ -static void __osm_ftree_fabric_dump(ftree_fabric_t * p_ftree) +static void fabric_dump(ftree_fabric_t * p_ftree) { uint32_t i; ftree_hca_t *p_hca; @@ -1102,7 +1093,7 @@ static void __osm_ftree_fabric_dump(ftree_fabric_t * p_ftree) for (p_hca = (ftree_hca_t *) cl_qmap_head(&p_ftree->hca_tbl); p_hca != (ftree_hca_t *) cl_qmap_end(&p_ftree->hca_tbl); p_hca = (ftree_hca_t *) cl_qmap_next(&p_hca->map_item)) { - __osm_ftree_hca_dump(p_ftree, p_hca); + hca_dump(p_ftree, p_hca); } for (i = 0; i <= p_ftree->max_switch_rank; i++) { @@ -1112,7 +1103,7 @@ static void __osm_ftree_fabric_dump(ftree_fabric_t * p_ftree) p_sw != (ftree_sw_t *) cl_qmap_end(&p_ftree->sw_tbl); p_sw = (ftree_sw_t *) cl_qmap_next(&p_sw->map_item)) { if (p_sw->rank == i) - __osm_ftree_sw_dump(p_ftree, p_sw); + sw_dump(p_ftree, p_sw); } } @@ -1120,11 +1111,11 @@ static void __osm_ftree_fabric_dump(ftree_fabric_t * p_ftree) " |---------------------------------------|\n" " |- Full fabric topology dump completed -|\n" " |---------------------------------------|\n\n"); -} /* __osm_ftree_fabric_dump() */ +} /* fabric_dump() */ /***************************************************/ -static void __osm_ftree_fabric_dump_general_info(IN ftree_fabric_t * p_ftree) +static void fabric_dump_general_info(IN ftree_fabric_t * p_ftree) { uint32_t i, j; ftree_sw_t *p_sw; @@ -1178,9 +1169,9 @@ static void __osm_ftree_fabric_dump_general_info(IN ftree_fabric_t * p_ftree) OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, " GUID: 0x%016" PRIx64 ", LID: %u, Index %s\n", - __osm_ftree_sw_get_guid_ho(p_sw), + sw_get_guid_ho(p_sw), cl_ntoh16(p_sw->base_lid), - __osm_ftree_tuple_to_str(p_sw->tuple)); + tuple_to_str(p_sw->tuple)); } OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, @@ -1189,19 +1180,16 @@ static void __osm_ftree_fabric_dump_general_info(IN ftree_fabric_t * p_ftree) OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, " GUID: 0x%016" PRIx64 ", LID: %u, Index %s\n", - __osm_ftree_sw_get_guid_ho(p_ftree-> - leaf_switches[i]), + sw_get_guid_ho(p_ftree->leaf_switches[i]), cl_ntoh16(p_ftree->leaf_switches[i]->base_lid), - __osm_ftree_tuple_to_str(p_ftree-> - leaf_switches[i]-> - tuple)); + tuple_to_str(p_ftree->leaf_switches[i]->tuple)); } } -} /* __osm_ftree_fabric_dump_general_info() */ +} /* fabric_dump_general_info() */ /***************************************************/ -static void __osm_ftree_fabric_dump_hca_ordering(IN ftree_fabric_t * p_ftree) +static void fabric_dump_hca_ordering(IN ftree_fabric_t * p_ftree) { ftree_hca_t *p_hca; ftree_sw_t *p_sw; @@ -1239,9 +1227,8 @@ static void __osm_ftree_fabric_dump_hca_ordering(IN ftree_fabric_t * p_ftree) p_hca = p_group_on_sw->remote_hca_or_sw.p_hca; p_group_on_hca = - __osm_ftree_hca_get_port_group_by_remote_lid(p_hca, - p_group_on_sw-> - base_lid); + hca_get_port_group_by_remote_lid(p_hca, + p_group_on_sw->base_lid); /* treat non-compute nodes as dummies */ if (!p_group_on_hca->is_cn) @@ -1263,49 +1250,49 @@ static void __osm_ftree_fabric_dump_hca_ordering(IN ftree_fabric_t * p_ftree) /* done going through all the leaf switches */ fclose(p_hca_ordering_file); -} /* __osm_ftree_fabric_dump_hca_ordering() */ +} /* fabric_dump_hca_ordering() */ /***************************************************/ static void -__osm_ftree_fabric_assign_tuple(IN ftree_fabric_t * p_ftree, - IN ftree_sw_t * p_sw, - IN ftree_tuple_t new_tuple) +fabric_assign_tuple(IN ftree_fabric_t * p_ftree, + IN ftree_sw_t * p_sw, + IN ftree_tuple_t new_tuple) { memcpy(p_sw->tuple, new_tuple, FTREE_TUPLE_LEN); - __osm_ftree_fabric_add_sw_by_tuple(p_ftree, p_sw); + fabric_add_sw_by_tuple(p_ftree, p_sw); } /***************************************************/ -static void __osm_ftree_fabric_assign_first_tuple(IN ftree_fabric_t * p_ftree, - IN ftree_sw_t * p_sw) +static void fabric_assign_first_tuple(IN ftree_fabric_t * p_ftree, + IN ftree_sw_t * p_sw) { uint8_t i; ftree_tuple_t new_tuple; - __osm_ftree_tuple_init(new_tuple); + tuple_init(new_tuple); new_tuple[0] = (uint8_t) p_sw->rank; for (i = 1; i <= p_sw->rank; i++) new_tuple[i] = 0; - __osm_ftree_fabric_assign_tuple(p_ftree, p_sw, new_tuple); + fabric_assign_tuple(p_ftree, p_sw, new_tuple); } /***************************************************/ static void -__osm_ftree_fabric_get_new_tuple(IN ftree_fabric_t * p_ftree, - OUT ftree_tuple_t new_tuple, - IN ftree_tuple_t from_tuple, - IN ftree_direction_t direction) +fabric_get_new_tuple(IN ftree_fabric_t * p_ftree, + OUT ftree_tuple_t new_tuple, + IN ftree_tuple_t from_tuple, + IN ftree_direction_t direction) { ftree_sw_t *p_sw; ftree_tuple_t temp_tuple; uint8_t var_index; uint8_t i; - __osm_ftree_tuple_init(new_tuple); + tuple_init(new_tuple); memcpy(temp_tuple, from_tuple, FTREE_TUPLE_LEN); if (direction == FTREE_DIRECTION_DOWN) { @@ -1318,7 +1305,7 @@ __osm_ftree_fabric_get_new_tuple(IN ftree_fabric_t * p_ftree, for (i = 0; i < 0xFF; i++) { temp_tuple[var_index] = i; - p_sw = __osm_ftree_fabric_get_sw_by_tuple(p_ftree, temp_tuple); + p_sw = fabric_get_sw_by_tuple(p_ftree, temp_tuple); if (p_sw == NULL) /* found free tuple */ break; } @@ -1329,35 +1316,32 @@ __osm_ftree_fabric_get_new_tuple(IN ftree_fabric_t * p_ftree, } memcpy(new_tuple, temp_tuple, FTREE_TUPLE_LEN); -} /* __osm_ftree_fabric_get_new_tuple() */ +} /* fabric_get_new_tuple() */ /***************************************************/ -static inline boolean_t __osm_ftree_fabric_roots_provided(IN ftree_fabric_t * - p_ftree) +static inline boolean_t fabric_roots_provided(IN ftree_fabric_t * p_ftree) { return (p_ftree->p_osm->subn.opt.root_guid_file != NULL); } /***************************************************/ -static inline boolean_t __osm_ftree_fabric_cns_provided(IN ftree_fabric_t * - p_ftree) +static inline boolean_t fabric_cns_provided(IN ftree_fabric_t * p_ftree) { return (p_ftree->p_osm->subn.opt.cn_guid_file != NULL); } /***************************************************/ -static inline boolean_t __osm_ftree_fabric_ios_provided(IN ftree_fabric_t * - p_ftree) +static inline boolean_t fabric_ios_provided(IN ftree_fabric_t * p_ftree) { return (p_ftree->p_osm->subn.opt.io_guid_file != NULL); } /***************************************************/ -static int __osm_ftree_fabric_mark_leaf_switches(IN ftree_fabric_t * p_ftree) +static int fabric_mark_leaf_switches(IN ftree_fabric_t * p_ftree) { ftree_sw_t *p_sw; ftree_hca_t *p_hca; @@ -1404,7 +1388,7 @@ static int __osm_ftree_fabric_mark_leaf_switches(IN ftree_fabric_t * p_ftree) "while FatTree leaf rank is %u\n", cl_ntoh64(p_hca->up_port_groups[i]-> port_guid), - __osm_ftree_sw_get_guid_ho(p_sw), + sw_get_guid_ho(p_sw), p_sw->rank, p_ftree->leaf_switch_rank); res = -1; goto Exit; @@ -1416,11 +1400,11 @@ static int __osm_ftree_fabric_mark_leaf_switches(IN ftree_fabric_t * p_ftree) Exit: OSM_LOG_EXIT(&p_ftree->p_osm->log); return res; -} /* __osm_ftree_fabric_mark_leaf_switches() */ +} /* fabric_mark_leaf_switches() */ /***************************************************/ -static void __osm_ftree_fabric_make_indexing(IN ftree_fabric_t * p_ftree) +static void fabric_make_indexing(IN ftree_fabric_t * p_ftree) { ftree_sw_t *p_remote_sw; ftree_sw_t *p_sw = NULL; @@ -1448,7 +1432,7 @@ static void __osm_ftree_fabric_make_indexing(IN ftree_fabric_t * p_ftree) /* Assign the first tuple to the switch that is used as BFS starting point. The tuple will be as follows: [rank].0.0.0... This fuction also adds the switch it into the switch_by_tuple table. */ - __osm_ftree_fabric_assign_first_tuple(p_ftree, p_sw); + fabric_assign_first_tuple(p_ftree, p_sw); OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, "Indexing starting point:\n" @@ -1456,8 +1440,8 @@ static void __osm_ftree_fabric_make_indexing(IN ftree_fabric_t * p_ftree) " - Switch index : %s\n" " - Node LID : %u\n" " - Node GUID : 0x%016" - PRIx64 "\n", p_sw->rank, __osm_ftree_tuple_to_str(p_sw->tuple), - cl_ntoh16(p_sw->base_lid), __osm_ftree_sw_get_guid_ho(p_sw)); + PRIx64 "\n", p_sw->rank, tuple_to_str(p_sw->tuple), + cl_ntoh16(p_sw->base_lid), sw_get_guid_ho(p_sw)); /* * Now run BFS and assign indexes to all switches @@ -1497,21 +1481,20 @@ static void __osm_ftree_fabric_make_indexing(IN ftree_fabric_t * p_ftree) p_remote_sw = p_sw->down_port_groups[i]->remote_hca_or_sw. p_sw; - if (__osm_ftree_tuple_assigned - (p_remote_sw->tuple)) { + if (tuple_assigned(p_remote_sw->tuple)) { /* this switch has been already indexed */ continue; } /* allocate new tuple */ - __osm_ftree_fabric_get_new_tuple(p_ftree, - new_tuple, - p_sw->tuple, - FTREE_DIRECTION_DOWN); + fabric_get_new_tuple(p_ftree, + new_tuple, + p_sw->tuple, + FTREE_DIRECTION_DOWN); /* Assign the new tuple to the remote switch. This fuction also adds the switch into the switch_by_tuple table. */ - __osm_ftree_fabric_assign_tuple(p_ftree, - p_remote_sw, - new_tuple); + fabric_assign_tuple(p_ftree, + p_remote_sw, + new_tuple); /* add the newly discovered switch to the BFS queue */ cl_list_insert_tail(&bfs_list, p_sw); @@ -1522,7 +1505,7 @@ static void __osm_ftree_fabric_make_indexing(IN ftree_fabric_t * p_ftree) qsort(p_sw->down_port_groups, /* array */ p_sw->down_port_groups_num, /* number of elements */ sizeof(ftree_port_group_t *), /* size of each element */ - __osm_ftree_compare_port_groups_by_remote_switch_index); /* comparator */ + compare_port_groups_by_remote_switch_index); /* comparator */ } /* Done indexing switches from ports that go down. @@ -1535,20 +1518,19 @@ static void __osm_ftree_fabric_make_indexing(IN ftree_fabric_t * p_ftree) p_remote_sw = p_sw->up_port_groups[i]->remote_hca_or_sw. p_sw; - if (__osm_ftree_tuple_assigned - (p_remote_sw->tuple)) + if (tuple_assigned(p_remote_sw->tuple)) continue; /* allocate new tuple */ - __osm_ftree_fabric_get_new_tuple(p_ftree, - new_tuple, - p_sw->tuple, - FTREE_DIRECTION_UP); + fabric_get_new_tuple(p_ftree, + new_tuple, + p_sw->tuple, + FTREE_DIRECTION_UP); /* Assign the new tuple to the remote switch. This fuction also adds the switch to the switch_by_tuple table. */ - __osm_ftree_fabric_assign_tuple(p_ftree, - p_remote_sw, - new_tuple); + fabric_assign_tuple(p_ftree, + p_remote_sw, + new_tuple); /* add the newly discovered switch to the BFS queue */ cl_list_insert_tail(&bfs_list, p_sw); } @@ -1558,7 +1540,7 @@ static void __osm_ftree_fabric_make_indexing(IN ftree_fabric_t * p_ftree) qsort(p_sw->up_port_groups, /* array */ p_sw->up_port_groups_num, /* number of elements */ sizeof(ftree_port_group_t *), /* size of each element */ - __osm_ftree_compare_port_groups_by_remote_switch_index); /* comparator */ + compare_port_groups_by_remote_switch_index); /* comparator */ } /* Done assigning indexes to all the switches that are directly connected to the current switch - go to the next switch in the BFS queue */ @@ -1566,12 +1548,11 @@ static void __osm_ftree_fabric_make_indexing(IN ftree_fabric_t * p_ftree) cl_list_destroy(&bfs_list); OSM_LOG_EXIT(&p_ftree->p_osm->log); -} /* __osm_ftree_fabric_make_indexing() */ +} /* fabric_make_indexing() */ /***************************************************/ -static int __osm_ftree_fabric_create_leaf_switch_array(IN ftree_fabric_t * - p_ftree) +static int fabric_create_leaf_switch_array(IN ftree_fabric_t * p_ftree) { ftree_sw_t *p_sw; ftree_sw_t *p_next_sw; @@ -1604,7 +1585,7 @@ static int __osm_ftree_fabric_create_leaf_switch_array(IN ftree_fabric_t * OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Adding switch 0x%" PRIx64 " to full leaf switch array\n", - __osm_ftree_sw_get_guid_ho(p_sw)); + sw_get_guid_ho(p_sw)); all_switches_at_leaf_level[all_leaf_idx++] = p_sw; } @@ -1614,7 +1595,7 @@ static int __osm_ftree_fabric_create_leaf_switch_array(IN ftree_fabric_t * qsort(all_switches_at_leaf_level, /* array */ all_leaf_idx, /* number of elements */ sizeof(ftree_sw_t *), /* size of each element */ - __osm_ftree_compare_switches_by_index); /* comparator */ + compare_switches_by_index); /* comparator */ /* check the first and the last REAL leaf (the one that has CNs) in the array of all the leafs */ @@ -1660,11 +1641,11 @@ static int __osm_ftree_fabric_create_leaf_switch_array(IN ftree_fabric_t * Exit: OSM_LOG_EXIT(&p_ftree->p_osm->log); return res; -} /* __osm_ftree_fabric_create_leaf_switch_array() */ +} /* fabric_create_leaf_switch_array() */ /***************************************************/ -static void __osm_ftree_fabric_set_max_cn_per_leaf(IN ftree_fabric_t * p_ftree) +static void fabric_set_max_cn_per_leaf(IN ftree_fabric_t * p_ftree) { unsigned i; unsigned j; @@ -1685,19 +1666,18 @@ static void __osm_ftree_fabric_set_max_cn_per_leaf(IN ftree_fabric_t * p_ftree) if (cns_on_this_leaf > p_ftree->max_cn_per_leaf) p_ftree->max_cn_per_leaf = cns_on_this_leaf; } -} /* __osm_ftree_fabric_set_max_cn_per_leaf() */ +} /* fabric_set_max_cn_per_leaf() */ /***************************************************/ -static boolean_t __osm_ftree_fabric_validate_topology(IN ftree_fabric_t * - p_ftree) +static boolean_t fabric_validate_topology(IN ftree_fabric_t * p_ftree) { ftree_port_group_t *p_group; ftree_port_group_t *p_ref_group; ftree_sw_t *p_sw; ftree_sw_t *p_next_sw; ftree_sw_t **reference_sw_arr; - uint16_t tree_rank = __osm_ftree_fabric_get_rank(p_ftree); + uint16_t tree_rank = fabric_get_rank(p_ftree); boolean_t res = TRUE; uint8_t i; @@ -1735,17 +1715,17 @@ static boolean_t __osm_ftree_fabric_validate_topology(IN ftree_fabric_t * ", LID %u, Index %s - %u groups\n" " GUID 0x%016" PRIx64 ", LID %u, Index %s - %u groups\n", - __osm_ftree_sw_get_guid_ho + sw_get_guid_ho (reference_sw_arr[p_sw->rank]), cl_ntoh16(reference_sw_arr[p_sw->rank]-> base_lid), - __osm_ftree_tuple_to_str + tuple_to_str (reference_sw_arr[p_sw->rank]->tuple), reference_sw_arr[p_sw->rank]-> up_port_groups_num, - __osm_ftree_sw_get_guid_ho(p_sw), + sw_get_guid_ho(p_sw), cl_ntoh16(p_sw->base_lid), - __osm_ftree_tuple_to_str(p_sw->tuple), + tuple_to_str(p_sw->tuple), p_sw->up_port_groups_num); res = FALSE; break; @@ -1762,17 +1742,17 @@ static boolean_t __osm_ftree_fabric_validate_topology(IN ftree_fabric_t * ", LID %u, Index %s - %u port groups\n" " GUID 0x%016" PRIx64 ", LID %u, Index %s - %u port groups\n", - __osm_ftree_sw_get_guid_ho + sw_get_guid_ho (reference_sw_arr[p_sw->rank]), cl_ntoh16(reference_sw_arr[p_sw->rank]-> base_lid), - __osm_ftree_tuple_to_str + tuple_to_str (reference_sw_arr[p_sw->rank]->tuple), reference_sw_arr[p_sw->rank]-> down_port_groups_num, - __osm_ftree_sw_get_guid_ho(p_sw), + sw_get_guid_ho(p_sw), cl_ntoh16(p_sw->base_lid), - __osm_ftree_tuple_to_str(p_sw->tuple), + tuple_to_str(p_sw->tuple), p_sw->down_port_groups_num); res = FALSE; break; @@ -1798,23 +1778,23 @@ static boolean_t __osm_ftree_fabric_validate_topology(IN ftree_fabric_t * " GUID 0x%016" PRIx64 ", LID %u, Index %s - %u ports\n", - __osm_ftree_sw_get_guid_ho + sw_get_guid_ho (reference_sw_arr [p_sw->rank]), cl_ntoh16 (reference_sw_arr [p_sw->rank]-> base_lid), - __osm_ftree_tuple_to_str + tuple_to_str (reference_sw_arr [p_sw->rank]->tuple), cl_ptr_vector_get_size (&p_ref_group->ports), - __osm_ftree_sw_get_guid_ho + sw_get_guid_ho (p_sw), cl_ntoh16(p_sw-> base_lid), - __osm_ftree_tuple_to_str + tuple_to_str (p_sw->tuple), cl_ptr_vector_get_size (&p_group->ports)); @@ -1845,23 +1825,23 @@ static boolean_t __osm_ftree_fabric_validate_topology(IN ftree_fabric_t * " GUID 0x%016" PRIx64 ", LID %u, Index %s - %u ports\n", - __osm_ftree_sw_get_guid_ho + sw_get_guid_ho (reference_sw_arr [p_sw->rank]), cl_ntoh16 (reference_sw_arr [p_sw->rank]-> base_lid), - __osm_ftree_tuple_to_str + tuple_to_str (reference_sw_arr [p_sw->rank]->tuple), cl_ptr_vector_get_size (&p_ref_group->ports), - __osm_ftree_sw_get_guid_ho + sw_get_guid_ho (p_sw), cl_ntoh16(p_sw-> base_lid), - __osm_ftree_tuple_to_str + tuple_to_str (p_sw->tuple), cl_ptr_vector_get_size (&p_group->ports)); @@ -1883,13 +1863,13 @@ static boolean_t __osm_ftree_fabric_validate_topology(IN ftree_fabric_t * free(reference_sw_arr); OSM_LOG_EXIT(&p_ftree->p_osm->log); return res; -} /* __osm_ftree_fabric_validate_topology() */ +} /* fabric_validate_topology() */ /*************************************************** ***************************************************/ -static void __osm_ftree_set_sw_fwd_table(IN cl_map_item_t * const p_map_item, - IN void *context) +static void set_sw_fwd_table(IN cl_map_item_t * const p_map_item, + IN void *context) { ftree_sw_t *p_sw = (ftree_sw_t * const)p_map_item; ftree_fabric_t *p_ftree = (ftree_fabric_t *) context; @@ -1916,15 +1896,15 @@ static void __osm_ftree_set_sw_fwd_table(IN cl_map_item_t * const p_map_item, */ static boolean_t -__osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, - IN ftree_sw_t * p_sw, - IN ftree_sw_t * p_prev_sw, - IN ib_net16_t target_lid, - IN uint8_t target_rank, - IN boolean_t is_real_lid, - IN boolean_t is_main_path, - IN uint8_t highest_rank_in_route, - IN uint16_t reverse_hops) +fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, + IN ftree_sw_t * p_sw, + IN ftree_sw_t * p_prev_sw, + IN ib_net16_t target_lid, + IN uint8_t target_rank, + IN boolean_t is_real_lid, + IN boolean_t is_main_path, + IN uint8_t highest_rank_in_route, + IN uint16_t reverse_hops) { ftree_sw_t *p_remote_sw; uint16_t ports_num; @@ -1994,9 +1974,9 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, "Loop of lenght %d in the fabric:\n " "Switch %s (LID %u) closes loop through switch %s (LID %u)\n", (p_remote_sw->rank - highest_rank_in_route) * 2, - __osm_ftree_tuple_to_str(p_remote_sw->tuple), + tuple_to_str(p_remote_sw->tuple), cl_ntoh16(p_group->base_lid), - __osm_ftree_tuple_to_str(p_sw->tuple), + tuple_to_str(p_sw->tuple), cl_ntoh16(p_group->remote_base_lid)); continue; } @@ -2041,7 +2021,7 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, p_min_port->remote_port_num; OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Switch %s: set path to CA LID %u through port %u\n", - __osm_ftree_tuple_to_str(p_remote_sw->tuple), + tuple_to_str(p_remote_sw->tuple), cl_ntoh16(target_lid), p_min_port->remote_port_num); @@ -2052,14 +2032,12 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, cl_ptr_vector_at(&p_group->ports, j, (void *)&p_port); - __osm_ftree_sw_set_hops(p_remote_sw, - cl_ntoh16(target_lid), - p_port->remote_port_num, - ((target_rank - - highest_rank_in_route) - + (p_remote_sw->rank - - highest_rank_in_route) - + reverse_hops * 2)); + sw_set_hops(p_remote_sw, + cl_ntoh16(target_lid), + p_port->remote_port_num, + ((target_rank - highest_rank_in_route) + + (p_remote_sw->rank - highest_rank_in_route) + + reverse_hops * 2)); } } @@ -2074,13 +2052,15 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, /* Recursion step: Assign upgoing ports by stepping down, starting on REMOTE switch */ - created_route |= __osm_ftree_fabric_route_upgoing_by_going_down(p_ftree, p_remote_sw, /* remote switch - used as a route-upgoing alg. start point */ - NULL, /* prev. position - NULL to mark that we went down and not up */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - is_real_lid, /* whether the target LID is real or dummy */ - is_main_path, /* whether this is path to HCA that should by tracked by counters */ - highest_rank_in_route, reverse_hops); /* highest visited point in the tree before going down */ + created_route |= + fabric_route_upgoing_by_going_down( + p_ftree, p_remote_sw, /* remote switch - used as a route-upgoing alg. start point */ + NULL, /* prev. position - NULL to mark that we went down and not up */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + is_real_lid, /* whether the target LID is real or dummy */ + is_main_path, /* whether this is path to HCA that should by tracked by counters */ + highest_rank_in_route, reverse_hops); /* highest visited point in the tree before going down */ } /* done scanning all the down-going port groups */ @@ -2093,7 +2073,7 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, 1) % p_sw->down_port_groups_num; return created_route; -} /* __osm_ftree_fabric_route_upgoing_by_going_down() */ +} /* fabric_route_upgoing_by_going_down() */ /***************************************************/ @@ -2109,15 +2089,15 @@ __osm_ftree_fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, */ static void -__osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, - IN ftree_sw_t * p_sw, - IN ftree_sw_t * p_prev_sw, - IN ib_net16_t target_lid, - IN uint8_t target_rank, - IN boolean_t is_real_lid, - IN boolean_t is_main_path, - IN uint16_t reverse_hop_credit, - IN uint16_t reverse_hops) +fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, + IN ftree_sw_t * p_sw, + IN ftree_sw_t * p_prev_sw, + IN ib_net16_t target_lid, + IN uint8_t target_rank, + IN boolean_t is_real_lid, + IN boolean_t is_main_path, + IN uint16_t reverse_hop_credit, + IN uint16_t reverse_hops) { ftree_sw_t *p_remote_sw; uint16_t ports_num; @@ -2132,14 +2112,14 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, CL_ASSERT(is_real_lid || is_main_path); /* Assign upgoing ports by stepping down, starting on THIS switch */ - __osm_ftree_fabric_route_upgoing_by_going_down(p_ftree, p_sw, /* local switch - used as a route-upgoing alg. start point */ - p_prev_sw, /* switch that we went up from (NULL means that we went down) */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - is_real_lid, /* whether this target LID is real or dummy */ - is_main_path, /* whether this path to HCA should by tracked by counters */ - p_sw->rank, /* the highest visited point in the tree before going down */ - reverse_hops); /* Number of reverse_hops done up to this point */ + fabric_route_upgoing_by_going_down(p_ftree, p_sw, /* local switch - used as a route-upgoing alg. start point */ + p_prev_sw, /* switch that we went up from (NULL means that we went down) */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + is_real_lid, /* whether this target LID is real or dummy */ + is_main_path, /* whether this path to HCA should by tracked by counters */ + p_sw->rank, /* the highest visited point in the tree before going down */ + reverse_hops); /* Number of reverse_hops done up to this point */ /* recursion stop condition - if it's a root switch, */ if (p_sw->rank == 0) { @@ -2161,14 +2141,15 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, continue; p_remote_sw = p_group->remote_hca_or_sw.p_sw; - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ - p_sw, /* this switch - prev. position switch for the function */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - is_real_lid, /* whether this target LID is real or dummy */ - is_main_path, /* whether this is path to HCA that should by tracked by counters */ - reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ - reverse_hops + 1); /* Number of reverse_hops done up to this point */ + fabric_route_downgoing_by_going_up( + p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + p_sw, /* this switch - prev. position switch for the function */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + is_real_lid, /* whether this target LID is real or dummy */ + is_main_path, /* whether this is path to HCA that should by tracked by counters */ + reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ + reverse_hops + 1); /* Number of reverse_hops done up to this point */ } } @@ -2249,8 +2230,8 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, " - Routing MAIN path for %s CA LID %u: %s --> %s\n", (is_real_lid) ? "real" : "DUMMY", cl_ntoh16(target_lid), - __osm_ftree_tuple_to_str(p_sw->tuple), - __osm_ftree_tuple_to_str(p_remote_sw->tuple)); + tuple_to_str(p_sw->tuple), + tuple_to_str(p_remote_sw->tuple)); } /* The number of downgoing routes is tracked in the p_group->counter_down p_port->counter_down counters of the @@ -2268,8 +2249,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, p_min_port->remote_port_num; OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Switch %s: set path to CA LID %u through port %u\n", - __osm_ftree_tuple_to_str(p_remote_sw-> - tuple), + tuple_to_str(p_remote_sw->tuple), cl_ntoh16(target_lid), p_min_port->remote_port_num); } @@ -2282,25 +2262,25 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, for (j = 0; j < ports_num; j++) { cl_ptr_vector_at(&p_min_group->ports, j, (void *)&p_port); - __osm_ftree_sw_set_hops(p_remote_sw, - cl_ntoh16(target_lid), - p_port->remote_port_num, - target_rank - - p_remote_sw->rank + - 2 * reverse_hops); + sw_set_hops(p_remote_sw, + cl_ntoh16(target_lid), + p_port->remote_port_num, + target_rank - p_remote_sw->rank + + 2 * reverse_hops); } } /* Recursion step: Assign downgoing ports by stepping up, starting on REMOTE switch. */ - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ - p_sw, /* this switch - prev. position switch for the function */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - is_real_lid, /* whether this target LID is real or dummy */ - is_main_path, /* whether this is path to HCA that should by tracked by counters */ - reverse_hop_credit, /* Remaining reverse_hops allowed */ - reverse_hops); /* Number of reverse_hops done up to this point */ + fabric_route_downgoing_by_going_up( + p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + p_sw, /* this switch - prev. position switch for the function */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + is_real_lid, /* whether this target LID is real or dummy */ + is_main_path, /* whether this is path to HCA that should by tracked by counters */ + reverse_hop_credit, /* Remaining reverse_hops allowed */ + reverse_hops); /* Number of reverse_hops done up to this point */ } /* we're done for the third case */ @@ -2352,8 +2332,8 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, " - Routing SECONDARY path for LID %u: %s --> %s\n", cl_ntoh16(target_lid), - __osm_ftree_tuple_to_str(p_sw->tuple), - __osm_ftree_tuple_to_str(p_remote_sw->tuple)); + tuple_to_str(p_sw->tuple), + tuple_to_str(p_remote_sw->tuple)); } /* Routing REAL lids on SECONDARY path means routing @@ -2373,24 +2353,24 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, for (j = 0; j < ports_num; j++) { cl_ptr_vector_at(&p_group->ports, j, (void *)&p_port); - __osm_ftree_sw_set_hops(p_remote_sw, - cl_ntoh16(target_lid), - p_port->remote_port_num, - target_rank - - p_remote_sw->rank + - 2 * reverse_hops); + sw_set_hops(p_remote_sw, + cl_ntoh16(target_lid), + p_port->remote_port_num, + target_rank - p_remote_sw->rank + + 2 * reverse_hops); } /* Recursion step: Assign downgoing ports by stepping up, starting on REMOTE switch. */ - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ - p_sw, /* this switch - prev. position switch for the function */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - TRUE, /* whether the target LID is real or dummy */ - FALSE, /* whether this is path to HCA that should by tracked by counters */ - reverse_hop_credit, /* Remaining reverse_hops allowed */ - reverse_hops); /* Number of reverse_hops done up to this point */ + fabric_route_downgoing_by_going_up( + p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + p_sw, /* this switch - prev. position switch for the function */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + TRUE, /* whether the target LID is real or dummy */ + FALSE, /* whether this is path to HCA that should by tracked by counters */ + reverse_hop_credit, /* Remaining reverse_hops allowed */ + reverse_hops); /* Number of reverse_hops done up to this point */ } /* If we don't have any reverse hop credits, we are done */ @@ -2411,17 +2391,18 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* Recursion step: Assign downgoing ports by stepping up, fter doing one step down starting on REMOTE switch. */ - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ - p_sw, /* this switch - prev. position switch for the function */ - target_lid, /* LID that we're routing to */ - target_rank, /* rank of the LID that we're routing to */ - TRUE, /* whether the target LID is real or dummy */ - TRUE, /* whether this is path to HCA that should by tracked by counters */ - reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ - reverse_hops + 1); /* Number of reverse_hops done up to this point */ + fabric_route_downgoing_by_going_up( + p_ftree, p_remote_sw, /* remote switch - used as a route-downgoing alg. next step point */ + p_sw, /* this switch - prev. position switch for the function */ + target_lid, /* LID that we're routing to */ + target_rank, /* rank of the LID that we're routing to */ + TRUE, /* whether the target LID is real or dummy */ + TRUE, /* whether this is path to HCA that should by tracked by counters */ + reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ + reverse_hops + 1); /* Number of reverse_hops done up to this point */ } -} /* ftree_fabric_route_downgoing_by_going_up() */ +} /* ftree_fabric_route_downgoing_by_going_up() */ /***************************************************/ @@ -2436,7 +2417,7 @@ __osm_ftree_fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, * call assign-down-going-port-by-ascending-up(FALSE,TRUE) on CURRENT switch */ -static void __osm_ftree_fabric_route_to_cns(IN ftree_fabric_t * p_ftree) +static void fabric_route_to_cns(IN ftree_fabric_t * p_ftree) { ftree_sw_t *p_sw; ftree_hca_t *p_hca; @@ -2471,9 +2452,8 @@ static void __osm_ftree_fabric_route_to_cns(IN ftree_fabric_t * p_ftree) continue; p_hca_port_group = - __osm_ftree_hca_get_port_group_by_remote_lid(p_hca, - p_leaf_port_group-> - base_lid); + hca_get_port_group_by_remote_lid(p_hca, + p_leaf_port_group->base_lid); CL_ASSERT(p_hca_port_group); /* work with this port group only if remote port is CN */ @@ -2490,25 +2470,25 @@ static void __osm_ftree_fabric_route_to_cns(IN ftree_fabric_t * p_ftree) OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Switch %s: set path to CN LID %u through port %u\n", - __osm_ftree_tuple_to_str(p_sw->tuple), + tuple_to_str(p_sw->tuple), cl_ntoh16(hca_lid), p_port->port_num); /* set local min hop table(LID) to route to the CA */ - __osm_ftree_sw_set_hops(p_sw, - cl_ntoh16(hca_lid), - p_port->port_num, 1); + sw_set_hops(p_sw, cl_ntoh16(hca_lid), + p_port->port_num, 1); /* Assign downgoing ports by stepping up. Since we're routing here only CNs, we're routing it as REAL LID and updating fat-tree balancing counters. */ - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_sw, /* local switch - used as a route-downgoing alg. start point */ - NULL, /* prev. position switch */ - hca_lid, /* LID that we're routing to */ - p_sw->rank + 1, /* rank of the LID that we're routing to */ - TRUE, /* whether this HCA LID is real or dummy */ - TRUE, /* whether this path to HCA should by tracked by counters */ - 0, /* Number of reverse hops allowed */ - 0); /* Number of reverse hops done yet */ + fabric_route_downgoing_by_going_up( + p_ftree, p_sw, /* local switch - used as a route-downgoing alg. start point */ + NULL, /* prev. position switch */ + hca_lid, /* LID that we're routing to */ + p_sw->rank + 1, /* rank of the LID that we're routing to */ + TRUE, /* whether this HCA LID is real or dummy */ + TRUE, /* whether this path to HCA should by tracked by counters */ + 0, /* Number of reverse hops allowed */ + 0); /* Number of reverse hops done yet */ /* count how many real targets have been routed from this leaf switch */ routed_targets_on_leaf++; @@ -2528,20 +2508,21 @@ static void __osm_ftree_fabric_route_to_cns(IN ftree_fabric_t * p_ftree) (p_ftree->max_cn_per_leaf - routed_targets_on_leaf); j++) { /* assign downgoing ports by stepping up */ - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_sw, /* local switch - used as a route-downgoing alg. start point */ - NULL, /* prev. position switch */ - 0, /* LID that we're routing to - ignored for dummy HCA */ - 0, /* rank of the LID that we're routing to - ignored for dummy HCA */ - FALSE, /* whether this HCA LID is real or dummy */ - TRUE, /* whether this path to HCA should by tracked by counters */ - 0, /* Number of reverse hops allowed */ - 0); /* Number of reverse hops done yet */ + fabric_route_downgoing_by_going_up( + p_ftree, p_sw, /* local switch - used as a route-downgoing alg. start point */ + NULL, /* prev. position switch */ + 0, /* LID that we're routing to - ignored for dummy HCA */ + 0, /* rank of the LID that we're routing to - ignored for dummy HCA */ + FALSE, /* whether this HCA LID is real or dummy */ + TRUE, /* whether this path to HCA should by tracked by counters */ + 0, /* Number of reverse hops allowed */ + 0); /* Number of reverse hops done yet */ } } } /* done going through all the leaf switches */ OSM_LOG_EXIT(&p_ftree->p_osm->log); -} /* __osm_ftree_fabric_route_to_cns() */ +} /* fabric_route_to_cns() */ /***************************************************/ @@ -2562,7 +2543,7 @@ static void __osm_ftree_fabric_route_to_cns(IN ftree_fabric_t * p_ftree) * counters will not affect CN-to-CN routing. */ -static void __osm_ftree_fabric_route_to_non_cns(IN ftree_fabric_t * p_ftree) +static void fabric_route_to_non_cns(IN ftree_fabric_t * p_ftree) { ftree_sw_t *p_sw; ftree_hca_t *p_hca; @@ -2603,32 +2584,33 @@ static void __osm_ftree_fabric_route_to_non_cns(IN ftree_fabric_t * p_ftree) OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Switch %s: set path to non-CN HCA LID %u through port %u\n", - __osm_ftree_tuple_to_str(p_sw->tuple), + tuple_to_str(p_sw->tuple), cl_ntoh16(hca_lid), port_num_on_switch); /* set local min hop table(LID) to route to the CA */ - __osm_ftree_sw_set_hops(p_sw, cl_ntoh16(hca_lid), - port_num_on_switch, /* port num */ - 1); /* hops */ + sw_set_hops(p_sw, cl_ntoh16(hca_lid), + port_num_on_switch, /* port num */ + 1); /* hops */ /* Assign downgoing ports by stepping up. We're routing REAL targets. They are not CNs and not included in the leafs array, but we treat them as MAIN path to allow load leveling, which means that the counters will be updated. */ - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_sw, /* local switch - used as a route-downgoing alg. start point */ - NULL, /* prev. position switch */ - hca_lid, /* LID that we're routing to */ - p_sw->rank + 1, /* rank of the LID that we're routing to */ - TRUE, /* whether this HCA LID is real or dummy */ - TRUE, /* whether this path to HCA should by tracked by counters */ - p_hca_port_group->is_io ? p_ftree->p_osm->subn.opt.max_reverse_hops : 0, /* Number or reverse hops allowed */ - 0); /* Number or reverse hops done yet */ + fabric_route_downgoing_by_going_up( + p_ftree, p_sw, /* local switch - used as a route-downgoing alg. start point */ + NULL, /* prev. position switch */ + hca_lid, /* LID that we're routing to */ + p_sw->rank + 1, /* rank of the LID that we're routing to */ + TRUE, /* whether this HCA LID is real or dummy */ + TRUE, /* whether this path to HCA should by tracked by counters */ + p_hca_port_group->is_io ? p_ftree->p_osm->subn.opt.max_reverse_hops : 0, /* Number or reverse hops allowed */ + 0); /* Number or reverse hops done yet */ } /* done with all the port groups of this HCA - go to next HCA */ } OSM_LOG_EXIT(&p_ftree->p_osm->log); -} /* __osm_ftree_fabric_route_to_non_cns() */ +} /* fabric_route_to_non_cns() */ /***************************************************/ @@ -2644,7 +2626,7 @@ static void __osm_ftree_fabric_route_to_non_cns(IN ftree_fabric_t * p_ftree) * - we should NOT update port counters */ -static void __osm_ftree_fabric_route_to_switches(IN ftree_fabric_t * p_ftree) +static void fabric_route_to_switches(IN ftree_fabric_t * p_ftree) { ftree_sw_t *p_sw; ftree_sw_t *p_next_sw; @@ -2661,31 +2643,32 @@ static void __osm_ftree_fabric_route_to_switches(IN ftree_fabric_t * p_ftree) OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Switch %s (LID %u): routing switch-to-switch paths\n", - __osm_ftree_tuple_to_str(p_sw->tuple), + tuple_to_str(p_sw->tuple), cl_ntoh16(p_sw->base_lid)); /* set min hop table of the switch to itself */ - __osm_ftree_sw_set_hops(p_sw, cl_ntoh16(p_sw->base_lid), - 0, /* port_num */ - 0); /* hops */ + sw_set_hops(p_sw, cl_ntoh16(p_sw->base_lid), + 0, /* port_num */ + 0); /* hops */ - __osm_ftree_fabric_route_downgoing_by_going_up(p_ftree, p_sw, /* local switch - used as a route-downgoing alg. start point */ - NULL, /* prev. position switch */ - p_sw->base_lid, /* LID that we're routing to */ - p_sw->rank, /* rank of the LID that we're routing to */ - TRUE, /* whether the target LID is a real or dummy */ - FALSE, /* whether this path to HCA should by tracked by counters */ - 0, /* Number of reverse hops allowed */ - 0); /* Number of reverse hops done yet */ + fabric_route_downgoing_by_going_up( + p_ftree, p_sw, /* local switch - used as a route-downgoing alg. start point */ + NULL, /* prev. position switch */ + p_sw->base_lid, /* LID that we're routing to */ + p_sw->rank, /* rank of the LID that we're routing to */ + TRUE, /* whether the target LID is a real or dummy */ + FALSE, /* whether this path to HCA should by tracked by counters */ + 0, /* Number of reverse hops allowed */ + 0); /* Number of reverse hops done yet */ } OSM_LOG_EXIT(&p_ftree->p_osm->log); -} /* __osm_ftree_fabric_route_to_switches() */ +} /* fabric_route_to_switches() */ /*************************************************** ***************************************************/ -static int __osm_ftree_fabric_populate_nodes(IN ftree_fabric_t * p_ftree) +static int fabric_populate_nodes(IN ftree_fabric_t * p_ftree) { osm_node_t *p_osm_node; osm_node_t *p_next_osm_node; @@ -2702,12 +2685,12 @@ static int __osm_ftree_fabric_populate_nodes(IN ftree_fabric_t * p_ftree) (osm_node_t *) cl_qmap_next(&p_osm_node->map_item); switch (osm_node_get_type(p_osm_node)) { case IB_NODE_TYPE_CA: - __osm_ftree_fabric_add_hca(p_ftree, p_osm_node); + fabric_add_hca(p_ftree, p_osm_node); break; case IB_NODE_TYPE_ROUTER: break; case IB_NODE_TYPE_SWITCH: - __osm_ftree_fabric_add_sw(p_ftree, p_osm_node->sw); + fabric_add_sw(p_ftree, p_osm_node->sw); break; default: OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_ERROR, "ERR AB0E: " @@ -2723,15 +2706,15 @@ static int __osm_ftree_fabric_populate_nodes(IN ftree_fabric_t * p_ftree) OSM_LOG_EXIT(&p_ftree->p_osm->log); return 0; -} /* __osm_ftree_fabric_populate_nodes() */ +} /* fabric_populate_nodes() */ /*************************************************** ***************************************************/ -static boolean_t __osm_ftree_sw_update_rank(IN ftree_sw_t * p_sw, +static boolean_t sw_update_rank(IN ftree_sw_t * p_sw, IN uint32_t new_rank) { - if (__osm_ftree_sw_ranked(p_sw) && p_sw->rank <= new_rank) + if (sw_ranked(p_sw) && p_sw->rank <= new_rank) return FALSE; p_sw->rank = new_rank; return TRUE; @@ -2741,8 +2724,8 @@ static boolean_t __osm_ftree_sw_update_rank(IN ftree_sw_t * p_sw, /***************************************************/ static void -__osm_ftree_rank_switches_from_leafs(IN ftree_fabric_t * p_ftree, - IN cl_list_t * p_ranking_bfs_list) +rank_switches_from_leafs(IN ftree_fabric_t * p_ftree, + IN cl_list_t * p_ranking_bfs_list) { ftree_sw_t *p_sw; ftree_sw_t *p_remote_sw; @@ -2770,17 +2753,15 @@ __osm_ftree_rank_switches_from_leafs(IN ftree_fabric_t * p_ftree, IB_NODE_TYPE_SWITCH) continue; - p_remote_sw = __osm_ftree_fabric_get_sw_by_guid(p_ftree, - osm_node_get_node_guid - (p_remote_node)); + p_remote_sw = fabric_get_sw_by_guid(p_ftree, + osm_node_get_node_guid(p_remote_node)); if (!p_remote_sw) { /* remote node is not a switch */ continue; } /* if needed, rank the remote switch and add it to the BFS list */ - if (__osm_ftree_sw_update_rank - (p_remote_sw, p_sw->rank + 1)) { + if (sw_update_rank(p_remote_sw, p_sw->rank + 1)) { max_rank = p_remote_sw->rank; cl_list_insert_tail(p_ranking_bfs_list, p_remote_sw); @@ -2791,14 +2772,14 @@ __osm_ftree_rank_switches_from_leafs(IN ftree_fabric_t * p_ftree, /* set FatTree maximal switch rank */ p_ftree->max_switch_rank = max_rank; -} /* __osm_ftree_rank_switches_from_leafs() */ +} /* rank_switches_from_leafs() */ /***************************************************/ static int -__osm_ftree_rank_leaf_switches(IN ftree_fabric_t * p_ftree, - IN ftree_hca_t * p_hca, - IN cl_list_t * p_ranking_bfs_list) +rank_leaf_switches(IN ftree_fabric_t * p_ftree, + IN ftree_hca_t * p_hca, + IN cl_list_t * p_ranking_bfs_list) { ftree_sw_t *p_sw; osm_node_t *p_osm_node = p_hca->p_osm_node; @@ -2825,7 +2806,7 @@ __osm_ftree_rank_leaf_switches(IN ftree_fabric_t * p_ftree, OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_ERROR, "ERR AB0F: " "CA conected directly to another CA: " "0x%016" PRIx64 " <---> 0x%016" PRIx64 "\n", - __osm_ftree_hca_get_guid_ho(p_hca), + hca_get_guid_ho(p_hca), cl_ntoh64(osm_node_get_node_guid (p_remote_osm_node))); res = -1; @@ -2853,16 +2834,14 @@ __osm_ftree_rank_leaf_switches(IN ftree_fabric_t * p_ftree, /* remote node is switch */ - p_sw = __osm_ftree_fabric_get_sw_by_guid(p_ftree, - osm_node_get_node_guid - (p_osm_port-> - p_remote_physp-> - p_node)); + p_sw = fabric_get_sw_by_guid(p_ftree, + osm_node_get_node_guid( + p_osm_port->p_remote_physp->p_node)); CL_ASSERT(p_sw); /* if needed, rank the remote switch and add it to the BFS list */ - if (!__osm_ftree_sw_update_rank(p_sw, 0)) + if (!sw_update_rank(p_sw, 0)) continue; OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Marking rank of switch that is directly connected to CA:\n" @@ -2871,8 +2850,8 @@ __osm_ftree_rank_leaf_switches(IN ftree_fabric_t * p_ftree, " - Switch guid: 0x%016" PRIx64 "\n" " - Switch LID : %u\n", - __osm_ftree_hca_get_guid_ho(p_hca), - __osm_ftree_sw_get_guid_ho(p_sw), + hca_get_guid_ho(p_hca), + sw_get_guid_ho(p_sw), cl_ntoh16(p_sw->base_lid)); cl_list_insert_tail(p_ranking_bfs_list, p_sw); } @@ -2880,12 +2859,12 @@ __osm_ftree_rank_leaf_switches(IN ftree_fabric_t * p_ftree, Exit: OSM_LOG_EXIT(&p_ftree->p_osm->log); return res; -} /* __osm_ftree_rank_leaf_switches() */ +} /* rank_leaf_switches() */ /***************************************************/ -static void __osm_ftree_sw_reverse_rank(IN cl_map_item_t * const p_map_item, - IN void *context) +static void sw_reverse_rank(IN cl_map_item_t * const p_map_item, + IN void *context) { ftree_fabric_t *p_ftree = (ftree_fabric_t *) context; ftree_sw_t *p_sw = (ftree_sw_t * const)p_map_item; @@ -2896,8 +2875,8 @@ static void __osm_ftree_sw_reverse_rank(IN cl_map_item_t * const p_map_item, ***************************************************/ static int -__osm_ftree_fabric_construct_hca_ports(IN ftree_fabric_t * p_ftree, - IN ftree_hca_t * p_hca) +fabric_construct_hca_ports(IN ftree_fabric_t * p_ftree, + IN ftree_hca_t * p_hca) { ftree_sw_t *p_remote_sw; osm_node_t *p_node = p_hca->p_osm_node; @@ -2958,14 +2937,13 @@ __osm_ftree_fabric_construct_hca_ports(IN ftree_fabric_t * p_ftree, /* remote node is switch */ p_remote_sw = - __osm_ftree_fabric_get_sw_by_guid(p_ftree, - remote_node_guid); + fabric_get_sw_by_guid(p_ftree, remote_node_guid); CL_ASSERT(p_remote_sw); /* If CN file is not supplied, then all the CAs considered as Compute Nodes. Otherwise all the CAs are not CNs, and only guids that are present in the CN file will be marked as compute nodes. */ - if (!__osm_ftree_fabric_cns_provided(p_ftree)) { + if (!fabric_cns_provided(p_ftree)) { is_cn = TRUE; } else { name_map_item_t *p_elem = @@ -2986,7 +2964,7 @@ __osm_ftree_fabric_construct_hca_ports(IN ftree_fabric_t * p_ftree, "Marking CN port GUID 0x%016" PRIx64 "\n", cl_ntoh64(osm_physp_get_port_guid(p_osm_port))); } else { - if (__osm_ftree_fabric_ios_provided(p_ftree)) { + if (fabric_ios_provided(p_ftree)) { name_map_item_t *p_elem = (name_map_item_t *) cl_qmap_get(&p_ftree->io_guid_tbl, @@ -3013,28 +2991,28 @@ __osm_ftree_fabric_construct_hca_ports(IN ftree_fabric_t * p_ftree, } } - __osm_ftree_hca_add_port(p_hca, /* local ftree_hca object */ - i, /* local port number */ - remote_port_num, /* remote port number */ - osm_node_get_base_lid(p_node, i), /* local lid */ - osm_node_get_base_lid(p_remote_node, 0), /* remote lid */ - osm_physp_get_port_guid(p_osm_port), /* local port guid */ - osm_physp_get_port_guid(p_remote_osm_port), /* remote port guid */ - remote_node_guid, /* remote node guid */ - remote_node_type, /* remote node type */ - (void *)p_remote_sw, /* remote ftree_hca/sw object */ - is_cn, is_io); /* whether this port is compute node */ + hca_add_port(p_hca, /* local ftree_hca object */ + i, /* local port number */ + remote_port_num, /* remote port number */ + osm_node_get_base_lid(p_node, i), /* local lid */ + osm_node_get_base_lid(p_remote_node, 0), /* remote lid */ + osm_physp_get_port_guid(p_osm_port), /* local port guid */ + osm_physp_get_port_guid(p_remote_osm_port), /* remote port guid */ + remote_node_guid, /* remote node guid */ + remote_node_type, /* remote node type */ + (void *)p_remote_sw, /* remote ftree_hca/sw object */ + is_cn, is_io); /* whether this port is compute node */ } Exit: return res; -} /* __osm_ftree_fabric_construct_hca_ports() */ +} /* fabric_construct_hca_ports() */ /*************************************************** ***************************************************/ -static int __osm_ftree_fabric_construct_sw_ports(IN ftree_fabric_t * p_ftree, - IN ftree_sw_t * p_sw) +static int fabric_construct_sw_ports(IN ftree_fabric_t * p_ftree, + IN ftree_sw_t * p_sw) { ftree_hca_t *p_remote_hca; ftree_sw_t *p_remote_sw; @@ -3069,7 +3047,7 @@ static int __osm_ftree_fabric_construct_sw_ports(IN ftree_fabric_t * p_ftree, OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Ignoring loopback on switch GUID 0x%016" PRIx64 ", LID %u, rank %u\n", - __osm_ftree_sw_get_guid_ho(p_sw), + sw_get_guid_ho(p_sw), cl_ntoh16(p_sw->base_lid), p_sw->rank); continue; @@ -3087,8 +3065,7 @@ static int __osm_ftree_fabric_construct_sw_ports(IN ftree_fabric_t * p_ftree, /* switch connected to hca */ p_remote_hca = - __osm_ftree_fabric_get_hca_by_guid(p_ftree, - remote_node_guid); + fabric_get_hca_by_guid(p_ftree, remote_node_guid); CL_ASSERT(p_remote_hca); p_remote_hca_or_sw = (void *)p_remote_hca; @@ -3102,8 +3079,7 @@ static int __osm_ftree_fabric_construct_sw_ports(IN ftree_fabric_t * p_ftree, /* switch connected to another switch */ p_remote_sw = - __osm_ftree_fabric_get_sw_by_guid(p_ftree, - remote_node_guid); + fabric_get_sw_by_guid(p_ftree, remote_node_guid); CL_ASSERT(p_remote_sw); p_remote_hca_or_sw = (void *)p_remote_sw; @@ -3117,9 +3093,9 @@ static int __osm_ftree_fabric_construct_sw_ports(IN ftree_fabric_t * p_ftree, " GUID 0x%016" PRIx64 ", LID %u, rank %u\n", p_sw->rank, p_remote_sw->rank, - __osm_ftree_sw_get_guid_ho(p_sw), + sw_get_guid_ho(p_sw), cl_ntoh16(p_sw->base_lid), p_sw->rank, - __osm_ftree_sw_get_guid_ho(p_remote_sw), + sw_get_guid_ho(p_remote_sw), cl_ntoh16(p_remote_sw->base_lid), p_remote_sw->rank); res = -1; @@ -3146,17 +3122,17 @@ static int __osm_ftree_fabric_construct_sw_ports(IN ftree_fabric_t * p_ftree, res = -1; goto Exit; } - __osm_ftree_sw_add_port(p_sw, /* local ftree_sw object */ - i, /* local port number */ - remote_port_num, /* remote port number */ - p_sw->base_lid, /* local lid */ - remote_base_lid, /* remote lid */ - osm_physp_get_port_guid(p_osm_port), /* local port guid */ - osm_physp_get_port_guid(p_remote_osm_port), /* remote port guid */ - remote_node_guid, /* remote node guid */ - remote_node_type, /* remote node type */ - p_remote_hca_or_sw, /* remote ftree_hca/sw object */ - direction); /* port direction (up or down) */ + sw_add_port(p_sw, /* local ftree_sw object */ + i, /* local port number */ + remote_port_num, /* remote port number */ + p_sw->base_lid, /* local lid */ + remote_base_lid, /* remote lid */ + osm_physp_get_port_guid(p_osm_port), /* local port guid */ + osm_physp_get_port_guid(p_remote_osm_port), /* remote port guid */ + remote_node_guid, /* remote node guid */ + remote_node_type, /* remote node type */ + p_remote_hca_or_sw, /* remote ftree_hca/sw object */ + direction); /* port direction (up or down) */ /* Track the max lid (in host order) that exists in the fabric */ if (cl_ntoh16(remote_base_lid) > p_ftree->lft_max_lid_ho) @@ -3165,7 +3141,7 @@ static int __osm_ftree_fabric_construct_sw_ports(IN ftree_fabric_t * p_ftree, Exit: return res; -} /* __osm_ftree_fabric_construct_sw_ports() */ +} /* fabric_construct_sw_ports() */ /*************************************************** ***************************************************/ @@ -3179,7 +3155,7 @@ static int rank_root_sw_by_guid(void *cxt, uint64_t guid, char *p) struct rank_root_cxt *c = cxt; ftree_sw_t *sw; - sw = __osm_ftree_fabric_get_sw_by_guid(c->fabric, cl_hton64(guid)); + sw = fabric_get_sw_by_guid(c->fabric, cl_hton64(guid)); if (!sw) { /* the specified root guid wasn't found in the fabric */ OSM_LOG(&c->fabric->p_osm->log, OSM_LOG_ERROR, "ERR AB24: " @@ -3195,7 +3171,7 @@ static int rank_root_sw_by_guid(void *cxt, uint64_t guid, char *p) return 0; } -static int __osm_ftree_fabric_rank_from_roots(IN ftree_fabric_t * p_ftree) +static int fabric_rank_from_roots(IN ftree_fabric_t * p_ftree) { struct rank_root_cxt context; osm_node_t *p_osm_node; @@ -3258,18 +3234,16 @@ static int __osm_ftree_fabric_rank_from_roots(IN ftree_fabric_t * p_ftree) IB_NODE_TYPE_SWITCH) continue; - p_remote_sw = __osm_ftree_fabric_get_sw_by_guid(p_ftree, - osm_node_get_node_guid - (p_remote_osm_node)); + p_remote_sw = fabric_get_sw_by_guid(p_ftree, + osm_node_get_node_guid(p_remote_osm_node)); CL_ASSERT(p_remote_sw); /* if needed, rank the remote switch and add it to the BFS list */ - if (__osm_ftree_sw_update_rank - (p_remote_sw, p_sw->rank + 1)) { + if (sw_update_rank(p_remote_sw, p_sw->rank + 1)) { OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Ranking switch 0x%" PRIx64 " with rank %u\n", - __osm_ftree_sw_get_guid_ho(p_remote_sw), + sw_get_guid_ho(p_remote_sw), p_remote_sw->rank); max_rank = p_remote_sw->rank; cl_list_insert_tail(&ranking_bfs_list, @@ -3289,12 +3263,12 @@ Exit: cl_list_destroy(&ranking_bfs_list); OSM_LOG_EXIT(&p_ftree->p_osm->log); return res; -} /* __osm_ftree_fabric_rank_from_roots() */ +} /* fabric_rank_from_roots() */ /*************************************************** ***************************************************/ -static int __osm_ftree_fabric_rank_from_hcas(IN ftree_fabric_t * p_ftree) +static int fabric_rank_from_hcas(IN ftree_fabric_t * p_ftree) { ftree_hca_t *p_hca; ftree_hca_t *p_next_hca; @@ -3312,7 +3286,7 @@ static int __osm_ftree_fabric_rank_from_hcas(IN ftree_fabric_t * p_ftree) while (p_next_hca != (ftree_hca_t *) cl_qmap_end(&p_ftree->hca_tbl)) { p_hca = p_next_hca; p_next_hca = (ftree_hca_t *) cl_qmap_next(&p_hca->map_item); - if (__osm_ftree_rank_leaf_switches + if (rank_leaf_switches (p_ftree, p_hca, &ranking_bfs_list) != 0) { res = -1; OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_ERROR, "ERR AB14: " @@ -3323,31 +3297,31 @@ static int __osm_ftree_fabric_rank_from_hcas(IN ftree_fabric_t * p_ftree) /* Now rank rest of the switches in the fabric, while the list already contains all the ranked leaf switches */ - __osm_ftree_rank_switches_from_leafs(p_ftree, &ranking_bfs_list); + rank_switches_from_leafs(p_ftree, &ranking_bfs_list); /* fix ranking of the switches by reversing the ranking direction */ - cl_qmap_apply_func(&p_ftree->sw_tbl, __osm_ftree_sw_reverse_rank, + cl_qmap_apply_func(&p_ftree->sw_tbl, sw_reverse_rank, (void *)p_ftree); Exit: cl_list_destroy(&ranking_bfs_list); OSM_LOG_EXIT(&p_ftree->p_osm->log); return res; -} /* __osm_ftree_fabric_rank_from_hcas() */ +} /* fabric_rank_from_hcas() */ /*************************************************** ***************************************************/ -static int __osm_ftree_fabric_rank(IN ftree_fabric_t * p_ftree) +static int fabric_rank(IN ftree_fabric_t * p_ftree) { int res = 0; OSM_LOG_ENTER(&p_ftree->p_osm->log); - if (__osm_ftree_fabric_roots_provided(p_ftree)) - res = __osm_ftree_fabric_rank_from_roots(p_ftree); + if (fabric_roots_provided(p_ftree)) + res = fabric_rank_from_roots(p_ftree); else - res = __osm_ftree_fabric_rank_from_hcas(p_ftree); + res = fabric_rank_from_hcas(p_ftree); if (res) goto Exit; @@ -3358,12 +3332,12 @@ static int __osm_ftree_fabric_rank(IN ftree_fabric_t * p_ftree) Exit: OSM_LOG_EXIT(&p_ftree->p_osm->log); return res; -} /* __osm_ftree_fabric_rank() */ +} /* fabric_rank() */ /*************************************************** ***************************************************/ -static void __osm_ftree_fabric_set_leaf_rank(IN ftree_fabric_t * p_ftree) +static void fabric_set_leaf_rank(IN ftree_fabric_t * p_ftree) { unsigned i; ftree_sw_t *p_sw; @@ -3372,7 +3346,7 @@ static void __osm_ftree_fabric_set_leaf_rank(IN ftree_fabric_t * p_ftree) OSM_LOG_ENTER(&p_ftree->p_osm->log); - if (!__osm_ftree_fabric_roots_provided(p_ftree)) { + if (!fabric_roots_provided(p_ftree)) { /* If root file is not provided, the fabric has to be pure fat-tree in terms of ranking. Thus, leaf switches rank is the max rank. */ p_ftree->leaf_switch_rank = p_ftree->max_switch_rank; @@ -3395,7 +3369,7 @@ static void __osm_ftree_fabric_set_leaf_rank(IN ftree_fabric_t * p_ftree) OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Selected CN port GUID 0x%" PRIx64 "\n", - __osm_ftree_hca_get_guid_ho(p_hca)); + hca_get_guid_ho(p_hca)); for (i = 0; (i < p_hca->up_port_groups_num) && (!p_hca->up_port_groups[i]->is_cn); i++) ; @@ -3406,19 +3380,19 @@ static void __osm_ftree_fabric_set_leaf_rank(IN ftree_fabric_t * p_ftree) p_sw = p_hca->up_port_groups[i]->remote_hca_or_sw.p_sw; OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Selected leaf switch GUID 0x%" PRIx64 ", rank %u\n", - __osm_ftree_sw_get_guid_ho(p_sw), p_sw->rank); + sw_get_guid_ho(p_sw), p_sw->rank); p_ftree->leaf_switch_rank = p_sw->rank; } OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_INFO, "FatTree leaf switch rank is %u\n", p_ftree->leaf_switch_rank); OSM_LOG_EXIT(&p_ftree->p_osm->log); -} /* __osm_ftree_fabric_set_leaf_rank() */ +} /* fabric_set_leaf_rank() */ /*************************************************** ***************************************************/ -static int __osm_ftree_fabric_populate_ports(IN ftree_fabric_t * p_ftree) +static int fabric_populate_ports(IN ftree_fabric_t * p_ftree) { ftree_hca_t *p_hca; ftree_hca_t *p_next_hca; @@ -3432,7 +3406,7 @@ static int __osm_ftree_fabric_populate_ports(IN ftree_fabric_t * p_ftree) while (p_next_hca != (ftree_hca_t *) cl_qmap_end(&p_ftree->hca_tbl)) { p_hca = p_next_hca; p_next_hca = (ftree_hca_t *) cl_qmap_next(&p_hca->map_item); - if (__osm_ftree_fabric_construct_hca_ports(p_ftree, p_hca) != 0) { + if (fabric_construct_hca_ports(p_ftree, p_hca) != 0) { res = -1; goto Exit; } @@ -3442,7 +3416,7 @@ static int __osm_ftree_fabric_populate_ports(IN ftree_fabric_t * p_ftree) while (p_next_sw != (ftree_sw_t *) cl_qmap_end(&p_ftree->sw_tbl)) { p_sw = p_next_sw; p_next_sw = (ftree_sw_t *) cl_qmap_next(&p_sw->map_item); - if (__osm_ftree_fabric_construct_sw_ports(p_ftree, p_sw) != 0) { + if (fabric_construct_sw_ports(p_ftree, p_sw) != 0) { res = -1; goto Exit; } @@ -3450,7 +3424,7 @@ static int __osm_ftree_fabric_populate_ports(IN ftree_fabric_t * p_ftree) Exit: OSM_LOG_EXIT(&p_ftree->p_osm->log); return res; -} /* __osm_ftree_fabric_populate_ports() */ +} /* fabric_populate_ports() */ /*************************************************** ***************************************************/ @@ -3469,13 +3443,13 @@ static int add_guid_item_to_map(void *cxt, uint64_t guid, char *p) return 0; } -static int __osm_ftree_fabric_read_guid_files(IN ftree_fabric_t * p_ftree) +static int fabric_read_guid_files(IN ftree_fabric_t * p_ftree) { int status = 0; OSM_LOG_ENTER(&p_ftree->p_osm->log); - if (__osm_ftree_fabric_cns_provided(p_ftree)) { + if (fabric_cns_provided(p_ftree)) { OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Fetching compute nodes from file %s\n", p_ftree->p_osm->subn.opt.cn_guid_file); @@ -3484,7 +3458,7 @@ static int __osm_ftree_fabric_read_guid_files(IN ftree_fabric_t * p_ftree) add_guid_item_to_map, &p_ftree->cn_guid_tbl)) { OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_ERROR, "ERR AB23: " - "Problem parsin CN guid file\n"); + "Problem parsing CN guid file\n"); status = -1; goto Exit; } @@ -3498,7 +3472,7 @@ static int __osm_ftree_fabric_read_guid_files(IN ftree_fabric_t * p_ftree) } - if (__osm_ftree_fabric_ios_provided(p_ftree)) { + if (fabric_ios_provided(p_ftree)) { OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, "Fetching I/O nodes from file %s\n", p_ftree->p_osm->subn.opt.io_guid_file); @@ -3507,7 +3481,7 @@ static int __osm_ftree_fabric_read_guid_files(IN ftree_fabric_t * p_ftree) add_guid_item_to_map, &p_ftree->io_guid_tbl)) { OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_ERROR, - "ERR AB23: " "Problem parsin I/O guid file\n"); + "ERR AB23: Problem parsing I/O guid file\n"); status = -1; goto Exit; } @@ -3523,19 +3497,19 @@ static int __osm_ftree_fabric_read_guid_files(IN ftree_fabric_t * p_ftree) Exit: OSM_LOG_EXIT(&p_ftree->p_osm->log); return status; -} /*__osm_ftree_fabric_read_guid_files() */ +} /*fabric_read_guid_files() */ /*************************************************** ***************************************************/ -static int __osm_ftree_construct_fabric(IN void *context) +static int construct_fabric(IN void *context) { ftree_fabric_t *p_ftree = context; int status = 0; OSM_LOG_ENTER(&p_ftree->p_osm->log); - __osm_ftree_fabric_clear(p_ftree); + fabric_clear(p_ftree); if (p_ftree->p_osm->subn.opt.lmc > 0) { osm_log(&p_ftree->p_osm->log, OSM_LOG_SYS, @@ -3572,7 +3546,7 @@ static int __osm_ftree_construct_fabric(IN void *context) OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, "Populating FatTree Switch and CA tables\n"); - if (__osm_ftree_fabric_populate_nodes(p_ftree) != 0) { + if (fabric_populate_nodes(p_ftree) != 0) { osm_log(&p_ftree->p_osm->log, OSM_LOG_SYS, "Fabric topology is not fat-tree - " "falling back to default routing\n"); @@ -3582,7 +3556,7 @@ static int __osm_ftree_construct_fabric(IN void *context) OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, "Reading guid files provided by user\n"); - if (__osm_ftree_fabric_read_guid_files(p_ftree) != 0) { + if (fabric_read_guid_files(p_ftree) != 0) { osm_log(&p_ftree->p_osm->log, OSM_LOG_SYS, "Failed reading guid files - " "falling back to default routing\n"); @@ -3604,7 +3578,7 @@ static int __osm_ftree_construct_fabric(IN void *context) We will be able to check leaf switches rank and the whole tree rank after filling ports and marking CNs. */ OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, "Ranking FatTree\n"); - if (__osm_ftree_fabric_rank(p_ftree) != 0) { + if (fabric_rank(p_ftree) != 0) { osm_log(&p_ftree->p_osm->log, OSM_LOG_SYS, "Failed ranking the tree\n"); status = -1; @@ -3618,7 +3592,7 @@ static int __osm_ftree_construct_fabric(IN void *context) that's how the port direction is determined. */ OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, "Populating CA & switch ports\n"); - if (__osm_ftree_fabric_populate_ports(p_ftree) != 0) { + if (fabric_populate_ports(p_ftree) != 0) { osm_log(&p_ftree->p_osm->log, OSM_LOG_SYS, "Fabric topology is not a fat-tree\n"); status = -1; @@ -3632,13 +3606,13 @@ static int __osm_ftree_construct_fabric(IN void *context) /* Now that the CA ports have been created and CNs were marked, we can complete the fabric ranking - set leaf switches rank. */ - __osm_ftree_fabric_set_leaf_rank(p_ftree); + fabric_set_leaf_rank(p_ftree); - if (__osm_ftree_fabric_get_rank(p_ftree) > FAT_TREE_MAX_RANK || - __osm_ftree_fabric_get_rank(p_ftree) < FAT_TREE_MIN_RANK) { + if (fabric_get_rank(p_ftree) > FAT_TREE_MAX_RANK || + fabric_get_rank(p_ftree) < FAT_TREE_MIN_RANK) { osm_log(&p_ftree->p_osm->log, OSM_LOG_SYS, "Fabric rank is %u (should be between %u and %u)\n", - __osm_ftree_fabric_get_rank(p_ftree), FAT_TREE_MIN_RANK, + fabric_get_rank(p_ftree), FAT_TREE_MIN_RANK, FAT_TREE_MAX_RANK); status = -1; goto Exit; @@ -3648,7 +3622,7 @@ static int __osm_ftree_construct_fabric(IN void *context) p_ftree->leaf_switch_rank and that are also connected to CNs. As a by-product, this function also runs basic topology validation - it checks that all the CNs are at the same rank. */ - if (__osm_ftree_fabric_mark_leaf_switches(p_ftree)) { + if (fabric_mark_leaf_switches(p_ftree)) { osm_log(&p_ftree->p_osm->log, OSM_LOG_SYS, "Fabric topology is not a fat-tree\n"); status = -1; @@ -3659,14 +3633,14 @@ static int __osm_ftree_construct_fabric(IN void *context) This function also sorts leaf switch array by the switch index, sorts all the port arrays of the indexed switches by remote switch index, and creates switch-by-tuple table (sw_by_tuple_tbl) */ - __osm_ftree_fabric_make_indexing(p_ftree); + fabric_make_indexing(p_ftree); /* Create leaf switch array sorted by index. This array contains switches with rank equal to p_ftree->leaf_switch_rank and that are also connected to CNs (REAL leafs), and it may contain switches at the same leaf rank w/o CNs, if this is the order of indexing. In any case, the first and the last switches in the array are REAL leafs. */ - if (__osm_ftree_fabric_create_leaf_switch_array(p_ftree)) { + if (fabric_create_leaf_switch_array(p_ftree)) { osm_log(&p_ftree->p_osm->log, OSM_LOG_SYS, "Fabric topology is not a fat-tree\n"); status = -1; @@ -3674,19 +3648,19 @@ static int __osm_ftree_construct_fabric(IN void *context) } /* calculate and set ftree.max_cn_per_leaf field */ - __osm_ftree_fabric_set_max_cn_per_leaf(p_ftree); + fabric_set_max_cn_per_leaf(p_ftree); /* print general info about fabric topology */ - __osm_ftree_fabric_dump_general_info(p_ftree); + fabric_dump_general_info(p_ftree); /* dump full tree topology */ if (osm_log_is_active(&p_ftree->p_osm->log, OSM_LOG_DEBUG)) - __osm_ftree_fabric_dump(p_ftree); + fabric_dump(p_ftree); /* the fabric is required to be PURE fat-tree only if the root guid file hasn't been provided by user */ - if (!__osm_ftree_fabric_roots_provided(p_ftree) && - !__osm_ftree_fabric_validate_topology(p_ftree)) { + if (!fabric_roots_provided(p_ftree) && + !fabric_validate_topology(p_ftree)) { osm_log(&p_ftree->p_osm->log, OSM_LOG_SYS, "Fabric topology is not a fat-tree\n"); status = -1; @@ -3701,7 +3675,7 @@ Exit: if (status != 0) { OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, "Clearing FatTree Fabric data structures\n"); - __osm_ftree_fabric_clear(p_ftree); + fabric_clear(p_ftree); } else p_ftree->fabric_built = TRUE; @@ -3713,12 +3687,12 @@ Exit: OSM_LOG_EXIT(&p_ftree->p_osm->log); return status; -} /* __osm_ftree_construct_fabric() */ +} /* construct_fabric() */ /*************************************************** ***************************************************/ -static int __osm_ftree_do_routing(IN void *context) +static int do_routing(IN void *context) { ftree_fabric_t *p_ftree = context; int status = 0; @@ -3735,22 +3709,22 @@ static int __osm_ftree_do_routing(IN void *context) OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, "Filling switch forwarding tables for Compute Nodes\n"); - __osm_ftree_fabric_route_to_cns(p_ftree); + fabric_route_to_cns(p_ftree); OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, "Filling switch forwarding tables for non-CN targets\n"); - __osm_ftree_fabric_route_to_non_cns(p_ftree); + fabric_route_to_non_cns(p_ftree); OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, "Filling switch forwarding tables for switch-to-switch paths\n"); - __osm_ftree_fabric_route_to_switches(p_ftree); + fabric_route_to_switches(p_ftree); /* for each switch, set its fwd table */ - cl_qmap_apply_func(&p_ftree->sw_tbl, __osm_ftree_set_sw_fwd_table, + cl_qmap_apply_func(&p_ftree->sw_tbl, set_sw_fwd_table, (void *)p_ftree); /* write out hca ordering file */ - __osm_ftree_fabric_dump_hca_ordering(p_ftree); + fabric_dump_hca_ordering(p_ftree); OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_VERBOSE, "FatTree routing is done\n"); @@ -3763,11 +3737,11 @@ Exit: /*************************************************** ***************************************************/ -static void __osm_ftree_delete(IN void *context) +static void delete(IN void *context) { if (!context) return; - __osm_ftree_fabric_destroy((ftree_fabric_t *) context); + fabric_destroy((ftree_fabric_t *) context); } /*************************************************** @@ -3775,16 +3749,16 @@ static void __osm_ftree_delete(IN void *context) int osm_ucast_ftree_setup(struct osm_routing_engine *r, osm_opensm_t * p_osm) { - ftree_fabric_t *p_ftree = __osm_ftree_fabric_create(); + ftree_fabric_t *p_ftree = fabric_create(); if (!p_ftree) return -1; p_ftree->p_osm = p_osm; r->context = (void *)p_ftree; - r->build_lid_matrices = __osm_ftree_construct_fabric; - r->ucast_build_fwd_tables = __osm_ftree_do_routing; - r->delete = __osm_ftree_delete; + r->build_lid_matrices = construct_fabric; + r->ucast_build_fwd_tables = do_routing; + r->delete = delete; return 0; } -- 1.5.1.4 From kliteyn at dev.mellanox.co.il Thu Mar 12 02:34:53 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Thu, 12 Mar 2009 11:34:53 +0200 Subject: [ofa-general] [PATCH] opensm/osm_ucast_ftree.c: some refactoring Message-ID: <49B8D73D.6060009@dev.mellanox.co.il> Hi Sasha, Some code refactoring: setting entry in the lid matrices should no longer be done per link, but per group, which means that lid matrix should be updated for all the ports that are connected to the same remote node. Signed-off-by: Yevgeny Kliteynik --- opensm/opensm/osm_ucast_ftree.c | 85 ++++++++++++++++----------------------- 1 files changed, 35 insertions(+), 50 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index 4e52d4b..10b3dcc 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -560,6 +560,9 @@ static ftree_sw_t *sw_create(IN ftree_fabric_t * p_ftree, /* initialize lft buffer */ memset(p_osm_sw->new_lft, OSM_NO_PATH, IB_LID_UCAST_END_HO + 1); + /* set min hop table of the switch to itself */ + osm_switch_set_hops(p_osm_sw, cl_ntoh16(p_sw->base_lid), 0, 0); + return p_sw; } /* sw_create() */ @@ -710,13 +713,23 @@ sw_add_port(IN ftree_sw_t * p_sw, /***************************************************/ -static inline cl_status_t +static int sw_set_hops(IN ftree_sw_t * p_sw, - IN uint16_t lid_ho, IN uint8_t port_num, + IN uint16_t lid_ho, + IN ftree_port_group_t * p_group, IN uint8_t hops) { - /* set local min hop table(LID) */ - return osm_switch_set_hops(p_sw->p_osm_sw, lid_ho, port_num, hops); + ftree_port_t * p_port; + uint8_t i, ports_num; + + ports_num = (uint8_t) cl_ptr_vector_get_size(&p_group->ports); + for (i = 0; i < ports_num; i++) { + cl_ptr_vector_at(&p_group->ports, i, (void *)&p_port); + if (osm_switch_set_hops(p_sw->p_osm_sw, lid_ho, + p_port->port_num, hops)) + return -1; + } + return 0; } /*************************************************** @@ -2027,19 +2040,11 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, /* On the remote switch that is pointed by the p_group, set hops for ALL the ports in the remote group. */ - - for (j = 0; j < ports_num; j++) { - cl_ptr_vector_at(&p_group->ports, j, - (void *)&p_port); - - sw_set_hops(p_remote_sw, - cl_ntoh16(target_lid), - p_port->remote_port_num, - ((target_rank - highest_rank_in_route) + - (p_remote_sw->rank - highest_rank_in_route) - + reverse_hops * 2)); - } - + sw_set_hops(p_remote_sw, + cl_ntoh16(target_lid), p_group, + ((target_rank - highest_rank_in_route) + + (p_remote_sw->rank - highest_rank_in_route) + + reverse_hops * 2)); } /* The number of upgoing routes is tracked in the @@ -2255,19 +2260,10 @@ fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, } /* On the remote switch that is pointed by the min_group, set hops for ALL the ports in the remote group. */ - - ports_num = - (uint16_t) cl_ptr_vector_get_size(&p_min_group-> - ports); - for (j = 0; j < ports_num; j++) { - cl_ptr_vector_at(&p_min_group->ports, j, - (void *)&p_port); - sw_set_hops(p_remote_sw, - cl_ntoh16(target_lid), - p_port->remote_port_num, - target_rank - p_remote_sw->rank + - 2 * reverse_hops); - } + sw_set_hops(p_remote_sw, + cl_ntoh16(target_lid), p_min_group, + target_rank - p_remote_sw->rank + + 2 * reverse_hops); } /* Recursion step: @@ -2348,17 +2344,9 @@ fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* On the remote switch that is pointed by the p_group, set hops for ALL the ports in the remote group. */ - - ports_num = (uint16_t) cl_ptr_vector_get_size(&p_group->ports); - for (j = 0; j < ports_num; j++) { - cl_ptr_vector_at(&p_group->ports, j, (void *)&p_port); - - sw_set_hops(p_remote_sw, - cl_ntoh16(target_lid), - p_port->remote_port_num, - target_rank - p_remote_sw->rank + - 2 * reverse_hops); - } + sw_set_hops(p_remote_sw, + cl_ntoh16(target_lid), p_group, + target_rank - p_remote_sw->rank + 2 * reverse_hops); /* Recursion step: Assign downgoing ports by stepping up, starting on REMOTE switch. */ @@ -2475,7 +2463,7 @@ static void fabric_route_to_cns(IN ftree_fabric_t * p_ftree) /* set local min hop table(LID) to route to the CA */ sw_set_hops(p_sw, cl_ntoh16(hca_lid), - p_port->port_num, 1); + p_leaf_port_group, 1); /* Assign downgoing ports by stepping up. Since we're routing here only CNs, we're routing it as REAL @@ -2550,6 +2538,7 @@ static void fabric_route_to_non_cns(IN ftree_fabric_t * p_ftree) ftree_hca_t *p_next_hca; ftree_port_t *p_hca_port; ftree_port_group_t *p_hca_port_group; + ftree_port_group_t *p_sw_port_group; ib_net16_t hca_lid; unsigned port_num_on_switch; unsigned i; @@ -2587,10 +2576,12 @@ static void fabric_route_to_non_cns(IN ftree_fabric_t * p_ftree) tuple_to_str(p_sw->tuple), cl_ntoh16(hca_lid), port_num_on_switch); + p_sw_port_group = sw_get_port_group_by_remote_lid( + p_sw, hca_lid, FTREE_DIRECTION_DOWN); + /* set local min hop table(LID) to route to the CA */ sw_set_hops(p_sw, cl_ntoh16(hca_lid), - port_num_on_switch, /* port num */ - 1); /* hops */ + p_sw_port_group, 1); /* Assign downgoing ports by stepping up. We're routing REAL targets. They are not CNs and not included @@ -2618,7 +2609,6 @@ static void fabric_route_to_non_cns(IN ftree_fabric_t * p_ftree) * Pseudo code: * foreach switch in fabric * obtain its LID - * set local LFT(LID) to port 0 * call assign-down-going-port-by-ascending-up(TRUE,FALSE) on CURRENT switch * * Routing to switch is similar to routing a REAL hca lid on SECONDARY path: @@ -2646,11 +2636,6 @@ static void fabric_route_to_switches(IN ftree_fabric_t * p_ftree) tuple_to_str(p_sw->tuple), cl_ntoh16(p_sw->base_lid)); - /* set min hop table of the switch to itself */ - sw_set_hops(p_sw, cl_ntoh16(p_sw->base_lid), - 0, /* port_num */ - 0); /* hops */ - fabric_route_downgoing_by_going_up( p_ftree, p_sw, /* local switch - used as a route-downgoing alg. start point */ NULL, /* prev. position switch */ -- 1.5.1.4 From ogerlitz at voltaire.com Thu Mar 12 02:43:19 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 12 Mar 2009 11:43:19 +0200 (IST) Subject: [ofa-general] Re: can't use perfquery from git against late kernels In-Reply-To: References: Message-ID: > I just case across a problem where with the latest management git - I can't use > perfquery against latest kernel (I also see the problem if running against the > drivers provided by ofed 1.4.x, but not against 1.3.y !) > > The problem doesn't exist for smpquery - and also it doesn't exist if I use > perfquery from 1.4.x / 1.3.y - so it seems something was broken for user space > GMPs between late kernels and latest mng git. Sorry, bit I was a bit wrong here - the problem does exist if running mng tree latest against 1.4.y / 1.3.x . I went back on the mng git to the opensm-3.2 branch (whose head is on 8989491f7bdba6c21de61c3e988c7d91d08e4254 Feb 26, 2009) and it works fine. I could do a bit more manual search on the mng commits - but I wasn't sure how to use the git tags - it seems that each library has its own tags enum sequence... it worked up till yesterday... Or. From tziporet at dev.mellanox.co.il Thu Mar 12 02:57:23 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Thu, 12 Mar 2009 11:57:23 +0200 Subject: [ofa-general] ***SPAM*** ib_mthca 0000:06:00.0: Catastrophic error detected: internal error In-Reply-To: <6e4f44220903111818r2d60baaes7bf485bea5641060@mail.gmail.com> References: <6e4f44220903111818r2d60baaes7bf485bea5641060@mail.gmail.com> Message-ID: <49B8DC83.9040302@mellanox.co.il> Phillip Wilson wrote: > I looked through the ofa-general mail archives and found this issue in > the November 6 through 10, 2008 archives. According to the mail > thread, the issue was fix by upgrading the firmware. I have the > latest posted 1.2.0 firmware, but I am running into the "ib_mthca: > Catastrophic error detected: internal error" issue. The cards are > reset when this issue occurs and the LIDs ( sm_lid, port_lid) are > reset to 0. > Please contact Mellanox support to get a FW with this bug fixed. This release is in RC mode now and has not been placed on our web yet. Tziporet From ogerlitz at voltaire.com Thu Mar 12 03:04:07 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 12 Mar 2009 12:04:07 +0200 (IST) Subject: [ofa-general] Re: can't use perfquery from git against late kernels In-Reply-To: References: Message-ID: > latest against 1.4.y / 1.3.x . I went back on the mng git to the opensm-3.2 branch > (whose head is on 8989491f7bdba6c21de61c3e988c7d91d08e4254 Feb 26, 2009) and it works fine OK, commit 229f17744c339ff2a95ad298e1ed76da6786b799 Mon Mar 9 "opensm: fix build warning with --disable-console-socket" works fine, so its one/more of this list commit 4ec6120d85c0ccfb4fb62cb8b50f27411c81e268 Author: Hal Rosenstock Date: Fri Mar 6 13:20:53 2009 -0500 libvendor/osm_vendor_mlx_dispatcher.c: Eliminate no longer needed osmv_mad_is_response Use ib_mad_is_response (which now supports trap repress) Signed-off-by: Hal Rosenstock Signed-off-by: Sasha Khapyorsky commit b639e64d39d11c08992d0726fa05244ba46a8253 Author: Hal Rosenstock Date: Fri Mar 6 13:19:48 2009 -0500 opensm: Handle trap repress on trap 144 generation Make trap generation sending be a transaction (expect a response) and handle incoming trap repress (notice attribute) ib_mad_is_response is now changed to better reflect responses (not just R bit in MAD header) Signed-off-by: Hal Rosenstock Signed-off-by: Sasha Khapyorsky commit b0342053ba4f1015ff521a4a81d322771c775d8c Author: Hal Rosenstock Date: Fri Mar 6 07:30:38 2009 -0500 Add pkey table support to osm_get_all_port_attr Only supported in osm_vendor_ibumad.c (separate patch for other vendor layers) Also, update applications using this (osmtest, opensm) Signed-off-by: Hal Rosenstock Signed-off-by: Sasha Khapyorsky commit 6462999ff0eb0369d6b0c2d9bda8c89de068066f Author: Hal Rosenstock Date: Mon Mar 9 06:01:17 2009 -0500 opensm/osm_inform.c: In __osm_send_report, make sure p_report_madw valid before using Signed-off-by: Hal Rosenstock Signed-off-by: Sasha Khapyorsky commit 82df4676ad8823caf8a1b496044127591926768c Author: Hal Rosenstock Date: Mon Mar 9 06:00:20 2009 -0500 opensm/osm_req.c: Shouldn't reveal port's MKey on Trap method Following IBA 1.2.1 o14-1 - SMA shall fill the M_Key field with zero. Signed-off-by: Hal Rosenstock Signed-off-by: Sasha Khapyorsky commit f5df8cd85697f7040154d8095d24bea50f5e4aef Author: David A. McMillen Date: Mon Mar 9 13:53:21 2009 -0700 infiniband-diags/ibnetdiscover: Apply --node-name-map remapping to names printed by --ports option Signed-off-by: David A. McMillen Signed-off-by: Ira Weiny Signed-off-by: Sasha Khapyorsky commit 5cb83223a12cf0680cf5df596280536c9e94cd62 Author: Hal Rosenstock Date: Fri Mar 6 10:25:54 2009 -0500 opensm/PerfMgr: A few more esp0 changes Also, some cosmetic changes Signed-off-by: Hal Rosenstock Signed-off-by: Sasha Khapyorsky commit b0d25a4f29017e4a48bd9b73b4013fa4e65d3795 Author: Sasha Khapyorsky Date: Mon Mar 9 18:22:30 2009 +0200 opensm/osm_console_io.c: move cio_close() function Move cio_close() and console_close() functions above cio_open() to prevent cbuild failures. Signed-off-by: Sasha Khapyorsky commit d006140ab66631443321a2a11ba20c3aed602a63 Author: Sasha Khapyorsky Date: Mon Mar 9 18:22:30 2009 +0200 opensm/console: move cio_open() function Move cio_open() function so it will seat under same #ifdef block with others cio_* stuff. Also remove unneeded parentheses and typo in comment fix. Signed-off-by: Sasha Khapyorsky commit 070649d6dc832edde9cc3a289109b39211d80d7d Author: Sasha Khapyorsky Date: Sat Mar 7 23:27:34 2009 +0200 libibmad: cleanup deprecated function use This cleans up libibmad internal stuff from using potentially deprecated functions, which may go away soon. Also removes DEPRECATED attribute from mad_register_port_client() - it doesn't touch any global data. Signed-off-by: Sasha Khapyorsky commit 8da0d508e6284dbde71830340e77db99389c6460 Author: Or Gerlitz Date: Mon Mar 9 18:01:02 2009 +0200 infiniband-diags/perfquery: add srcport param Add srcport param for the PortXmtDataSL/PortRcvDataSL calls made to libibmad Signed-off-by: Or Gerlitz Signed-off-by: Sasha Khapyorsky From vlad at lists.openfabrics.org Thu Mar 12 03:18:41 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Thu, 12 Mar 2009 03:18:41 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090312-0200 daily build status Message-ID: <20090312101841.61E2EE61102@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From hal.rosenstock at gmail.com Thu Mar 12 03:28:17 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 12 Mar 2009 06:28:17 -0400 Subject: [ofa-general] can't use perfquery from git against late kernels In-Reply-To: References: Message-ID: On Thu, Mar 12, 2009 at 5:18 AM, Or Gerlitz wrote: > Sasha, > > I just case across a problem where with the latest management git - I can't use > perfquery against latest kernel (I also see the problem if running against the > drivers provided by ofed 1.4.x, but not against 1.3.y !) > > The problem doesn't exist for smpquery - and also it doesn't exist if I use > perfquery from 1.4.x / 1.3.y - so it seems something was broken for user space > GMPs between late kernels and latest mng git. > > --> run perfquery of 1.4.x > > # rpm -qf /usr/sbin/perfquery > infiniband-diags-1.4.3_20081127_a92a5e5-1.ofed1.4.rc6 > > # /usr/sbin/perfquery 4 1 > # Port counters: Lid 4 port 1 > PortSelect:......................1 > [... and so on all counters - works fine ...] > > --> run perfquery built from latest git > > # /home/ogerlitz/ib-mng/sbin/perfquery 4 1 > ibwarn: [8278] mad_rpc: _do_madrpc failed; dport (Lid 4) > /home/ogerlitz/ib-mng/sbin/perfquery: iberror: failed: classportinfo query > > running with -ddd (attached the two runs) - it seems that the > there's no response on the classportinfo PM mad. Looks like QKey is wrong (no longer being set ?) in request MAD. -- Hal > Or. > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hal.rosenstock at gmail.com Thu Mar 12 03:30:10 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 12 Mar 2009 06:30:10 -0400 Subject: ***SPAM*** Re: [ofa-general] Re: can't use perfquery from git against late kernels In-Reply-To: References: Message-ID: On Thu, Mar 12, 2009 at 6:04 AM, Or Gerlitz wrote: >> latest against 1.4.y / 1.3.x . I went back on the mng git to the opensm-3.2 branch >> (whose head is on 8989491f7bdba6c21de61c3e988c7d91d08e4254 Feb 26, 2009) and it works fine > > OK, commit 229f17744c339ff2a95ad298e1ed76da6786b799 Mon Mar 9 "opensm: fix > build warning with --disable-console-socket" works fine, so its one/more of this list > > > commit 4ec6120d85c0ccfb4fb62cb8b50f27411c81e268 > Author: Hal Rosenstock > Date:   Fri Mar 6 13:20:53 2009 -0500 > >    libvendor/osm_vendor_mlx_dispatcher.c: Eliminate no longer needed osmv_mad_is_response > >    Use ib_mad_is_response (which now supports trap repress) > >    Signed-off-by: Hal Rosenstock >    Signed-off-by: Sasha Khapyorsky > > > commit b639e64d39d11c08992d0726fa05244ba46a8253 > Author: Hal Rosenstock > Date:   Fri Mar 6 13:19:48 2009 -0500 > >    opensm: Handle trap repress on trap 144 generation > >    Make trap generation sending be a transaction (expect a response) >    and handle incoming trap repress (notice attribute) >    ib_mad_is_response is now changed to better reflect responses >    (not just R bit in MAD header) > >    Signed-off-by: Hal Rosenstock >    Signed-off-by: Sasha Khapyorsky > > > commit b0342053ba4f1015ff521a4a81d322771c775d8c > Author: Hal Rosenstock > Date:   Fri Mar 6 07:30:38 2009 -0500 > >    Add pkey table support to osm_get_all_port_attr > >    Only supported in osm_vendor_ibumad.c (separate patch for other >    vendor layers) >    Also, update applications using this (osmtest, opensm) > >    Signed-off-by: Hal Rosenstock >    Signed-off-by: Sasha Khapyorsky > > > commit 6462999ff0eb0369d6b0c2d9bda8c89de068066f > Author: Hal Rosenstock > Date:   Mon Mar 9 06:01:17 2009 -0500 > >    opensm/osm_inform.c: In __osm_send_report, make sure p_report_madw valid before using > >    Signed-off-by: Hal Rosenstock >    Signed-off-by: Sasha Khapyorsky > > > commit 82df4676ad8823caf8a1b496044127591926768c > Author: Hal Rosenstock > Date:   Mon Mar 9 06:00:20 2009 -0500 > >    opensm/osm_req.c: Shouldn't reveal port's MKey on Trap method > >    Following IBA 1.2.1 o14-1 - SMA shall fill the M_Key field with zero. > >    Signed-off-by: Hal Rosenstock >    Signed-off-by: Sasha Khapyorsky > > > commit f5df8cd85697f7040154d8095d24bea50f5e4aef > Author: David A. McMillen > Date:   Mon Mar 9 13:53:21 2009 -0700 > >    infiniband-diags/ibnetdiscover: Apply --node-name-map remapping to names printed by --ports option > >    Signed-off-by: David A. McMillen >    Signed-off-by: Ira Weiny >    Signed-off-by: Sasha Khapyorsky > > > commit 5cb83223a12cf0680cf5df596280536c9e94cd62 > Author: Hal Rosenstock > Date:   Fri Mar 6 10:25:54 2009 -0500 > >    opensm/PerfMgr: A few more esp0 changes > >    Also, some cosmetic changes > >    Signed-off-by: Hal Rosenstock >    Signed-off-by: Sasha Khapyorsky > > > commit b0d25a4f29017e4a48bd9b73b4013fa4e65d3795 > Author: Sasha Khapyorsky > Date:   Mon Mar 9 18:22:30 2009 +0200 > >    opensm/osm_console_io.c: move cio_close() function > >    Move cio_close() and console_close() functions above cio_open() to >    prevent cbuild failures. > >    Signed-off-by: Sasha Khapyorsky > > > commit d006140ab66631443321a2a11ba20c3aed602a63 > Author: Sasha Khapyorsky > Date:   Mon Mar 9 18:22:30 2009 +0200 > >    opensm/console: move cio_open() function > >    Move cio_open() function so it will seat under same #ifdef block with >    others cio_* stuff. Also remove unneeded parentheses and typo in comment >    fix. > >    Signed-off-by: Sasha Khapyorsky > Does it work after the commit below ? > commit 070649d6dc832edde9cc3a289109b39211d80d7d > Author: Sasha Khapyorsky > Date:   Sat Mar 7 23:27:34 2009 +0200 > >    libibmad: cleanup deprecated function use > >    This cleans up libibmad internal stuff from using potentially deprecated >    functions, which may go away soon. > >    Also removes DEPRECATED attribute from mad_register_port_client() - it >    doesn't touch any global data. > >    Signed-off-by: Sasha Khapyorsky > > > commit 8da0d508e6284dbde71830340e77db99389c6460 > Author: Or Gerlitz > Date:   Mon Mar 9 18:01:02 2009 +0200 > >    infiniband-diags/perfquery: add srcport param > >    Add srcport param for the PortXmtDataSL/PortRcvDataSL calls made to libibmad > >    Signed-off-by: Or Gerlitz >    Signed-off-by: Sasha Khapyorsky > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hal.rosenstock at gmail.com Thu Mar 12 03:30:49 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 12 Mar 2009 06:30:49 -0400 Subject: ***SPAM*** Re: [ofa-general] ***SPAM*** [PATCH] infiniband-diags/mcm_rereg_test.c: Add missing mad_rpc_close_port call In-Reply-To: <0DE999D9A91E4CCBA6714985A0211242@amr.corp.intel.com> References: <20090311135320.GA29870@comcast.net> <20090311090337.ec293807.weiny2@llnl.gov> <0DE999D9A91E4CCBA6714985A0211242@amr.corp.intel.com> Message-ID: On Wed, Mar 11, 2009 at 4:54 PM, Sean Hefty wrote: >>>> +#if 1 >>>> +     mad_rpc_close_port(srcport); >>>> +#else >>>>       umad_unregister(port, agent); >>>>       umad_close_port(port); >>>>       umad_done(); >>>> +#endif >>> >>> Why use "#if 1" here?  Why not just replace the umad calls with >>> mad_rpc_close_port? >> >>Just consistency with what's been done in other places in that file. >>Other similar places are ifdef'd so I kept that. > > My vote is to not add an #if 1 and keep dead code around.  If #if 1 appears in > other places, we should probably remove that dead code as well. Someone made the decision to want to be able to switch back and forth earlier. This should be directed to them. It's certainly easy to eliminate the old code. From ogerlitz at Voltaire.com Thu Mar 12 05:03:39 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Thu, 12 Mar 2009 14:03:39 +0200 Subject: [ofa-general] can't use perfquery from git against late kernels In-Reply-To: References: Message-ID: <49B8FA1B.1030800@Voltaire.com> Hal Rosenstock wrote: > Looks like QKey is wrong (no longer being set ?) in request MAD. Thanks for looking into that, grepping for qkey on both outputs doesn't show any difference both use 0x80010000 ?! Or. # grep -i qkey perfquery-* perfquery-git-latest-4ec6120d85c0ccfb4fb62cb8b50f27411c81e268:ibwarn: [8304] umad_set_addr: umad 0x7fff04a01bc0 dlid 4 dqp 1 sl 0, qkey 80010000 perfquery-git-latest-4ec6120d85c0ccfb4fb62cb8b50f27411c81e268:ibwarn: [8304] umad_addr_dump: qpn 1 qkey 0x80010000 lid 4 sl 0 perfquery-git-latest-4ec6120d85c0ccfb4fb62cb8b50f27411c81e268:ibwarn: [8304] umad_addr_dump: qpn 1 qkey 0x80010000 lid 4 sl 0 perfquery-git-latest-4ec6120d85c0ccfb4fb62cb8b50f27411c81e268:ibwarn: [8304] umad_addr_dump: qpn 1 qkey 0x80010000 lid 4 sl 0 perfquery-infiniband-diags-1.4.3_20081127_a92a5e5-1.ofed1.4.rc6:ibwarn: [8273] umad_set_addr: umad 0x7fffe8c9a620 dlid 4 dqp 1 sl 0, qkey 80010000 perfquery-infiniband-diags-1.4.3_20081127_a92a5e5-1.ofed1.4.rc6:ibwarn: [8273] umad_addr_dump: qpn 1 qkey 0x80010000 lid 0x4 sl 0 perfquery-infiniband-diags-1.4.3_20081127_a92a5e5-1.ofed1.4.rc6:ibwarn: [8273] umad_set_addr: umad 0x7fffe8c9a1d0 dlid 4 dqp 1 sl 0, qkey 80010000 perfquery-infiniband-diags-1.4.3_20081127_a92a5e5-1.ofed1.4.rc6:ibwarn: [8273] umad_addr_dump: qpn 1 qkey 0x80010000 lid 0x4 sl 0 From kliteyn at dev.mellanox.co.il Thu Mar 12 05:02:32 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Thu, 12 Mar 2009 14:02:32 +0200 Subject: [ofa-general] [PATCH] opensm/osm_subnet.c: fixing compiler warnings Message-ID: <49B8F9D8.9090506@dev.mellanox.co.il> Hi Sasha, There are couple of meaningless CL_ASSERT() lines in the parsing functions that are checking the parsed value. The compiler says that "comparison is always true due to limited range of data type". So either the 'val' type should be more general when parsing it, or the CL_ASSERT() lines are not needed. The following patch removes them. Signed-off-by: Yevgeny Kliteynik --- opensm/opensm/osm_subnet.c | 2 -- 1 files changed, 0 insertions(+), 2 deletions(-) diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index 4ab440b..ec15f8a 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -218,7 +218,6 @@ static void opts_parse_net16(IN osm_subn_t *p_subn, IN char *p_key, uint16_t *p_val1 = p_v1, *p_val2 = p_v2; uint16_t val = strtoul(p_val_str, NULL, 0); - CL_ASSERT(val < 0x10000); if (cl_hton16(val) != *p_val1) { log_config_value(p_key, "0x%04x", val); if (pfn) @@ -234,7 +233,6 @@ static void opts_parse_uint8(IN osm_subn_t *p_subn, IN char *p_key, uint8_t *p_val1 = p_v1, *p_val2 = p_v2; uint8_t val = strtoul(p_val_str, NULL, 0); - CL_ASSERT(val < 0x100); if (val != *p_val1) { log_config_value(p_key, "%u", val); if (pfn) -- 1.5.1.4 From sashak at voltaire.com Thu Mar 12 05:10:04 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 14:10:04 +0200 Subject: [ofa-general] [PATCH] libibmad/rpc: fix class registration bug [was: Re: can't use perfquery from git against late kernels] In-Reply-To: References: Message-ID: <20090312121004.GA6766@sashak.voltaire.com> Fix class registration bug in mad_rpc_open_port(). Now it uses mad_register_client_via() (and not mad_register_port_client()) which returns status value and not an agent value - so don't overwrite by zero agent number in port's class_agent array. Signed-off-by: Sasha Khapyorsky --- Hi Or, On 12:04 Thu 12 Mar , Or Gerlitz wrote: > > commit 070649d6dc832edde9cc3a289109b39211d80d7d > Author: Sasha Khapyorsky > Date: Sat Mar 7 23:27:34 2009 +0200 > > libibmad: cleanup deprecated function use > > This cleans up libibmad internal stuff from using potentially deprecated > functions, which may go away soon. > > Also removes DEPRECATED attribute from mad_register_port_client() - it > doesn't touch any global data. > > Signed-off-by: Sasha Khapyorsky This commit breaks tools which use non-SM class (such as perfquery). There is the fix, I will commit soon. Sasha libibmad/src/rpc.c | 4 +--- 1 files changed, 1 insertions(+), 3 deletions(-) diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c index 233874b..87f0cad 100644 --- a/libibmad/src/rpc.c +++ b/libibmad/src/rpc.c @@ -361,12 +361,11 @@ struct ibmad_port *mad_rpc_open_port(char *dev_name, int dev_port, while (num_classes--) { uint8_t rmpp_version = 0; int mgmt = *mgmt_classes++; - int agent; if (mgmt == IB_SA_CLASS) rmpp_version = 1; if (mgmt < 0 || mgmt >= MAX_CLASS || - (agent = mad_register_client_via(mgmt, rmpp_version, p)) < 0) { + mad_register_client_via(mgmt, rmpp_version, p) < 0) { IBWARN("client_register for mgmt %d failed", mgmt); if (!errno) errno = EINVAL; @@ -374,7 +373,6 @@ struct ibmad_port *mad_rpc_open_port(char *dev_name, int dev_port, free(p); return NULL; } - p->class_agents[mgmt] = agent; } return p; -- 1.6.1.2.319.gbd9e From nicolas.morey-chaisemartin at ext.bull.net Thu Mar 12 05:06:19 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Thu, 12 Mar 2009 13:06:19 +0100 Subject: [ofa-general] [PATCH] opensm/osm_ucast_ftree.c: some refactoring In-Reply-To: <49B8D73D.6060009@dev.mellanox.co.il> References: <49B8D73D.6060009@dev.mellanox.co.il> Message-ID: <49B8FABB.3080402@ext.bull.net> Hi, Are you sure of this? Seems to me as if there is a problem Yevgeny Kliteynik wrote: > @@ -710,13 +713,23 @@ sw_add_port(IN ftree_sw_t * p_sw, > > /***************************************************/ > > -static inline cl_status_t > +static int > sw_set_hops(IN ftree_sw_t * p_sw, > - IN uint16_t lid_ho, IN uint8_t port_num, > + IN uint16_t lid_ho, > + IN ftree_port_group_t * p_group, > IN uint8_t hops) > { > - /* set local min hop table(LID) */ > - return osm_switch_set_hops(p_sw->p_osm_sw, lid_ho, port_num, hops); > + ftree_port_t * p_port; > + uint8_t i, ports_num; > + > + ports_num = (uint8_t) cl_ptr_vector_get_size(&p_group->ports); > + for (i = 0; i < ports_num; i++) { > + cl_ptr_vector_at(&p_group->ports, i, (void *)&p_port); > + if (osm_switch_set_hops(p_sw->p_osm_sw, lid_ho, > + p_port->port_num, hops)) > + return -1; > + } > + return 0; > } > Function is setting hops value for the origin ports of the given port group on the given switch > /*************************************************** > @@ -2027,19 +2040,11 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, > > /* On the remote switch that is pointed by the p_group, > set hops for ALL the ports in the remote group. */ > - > - for (j = 0; j < ports_num; j++) { > - cl_ptr_vector_at(&p_group->ports, j, > - (void *)&p_port); > - > - sw_set_hops(p_remote_sw, > - cl_ntoh16(target_lid), > - p_port->remote_port_num, > - ((target_rank - highest_rank_in_route) + > - (p_remote_sw->rank - highest_rank_in_route) > - + reverse_hops * 2)); > - } > - > + sw_set_hops(p_remote_sw, > + cl_ntoh16(target_lid), p_group, > + ((target_rank - highest_rank_in_route) + > + (p_remote_sw->rank - highest_rank_in_route) + > + reverse_hops * 2)); > } Here you call the function wth the remote switch and a local port group. So if I'm write the port num you use on the function are the port num on p_sw and not p_remote_sw... The remote port group should be pass to this function but I don't think we have an easy way to get it. Other solution could be to factorize the code in another function which would call sw_set_hops and provide it the right port num. Anyway, how is the hops table used? In the ftree algorithm, it seems we only use the min value through osm_switch_get_least_hops so we don't use the fact that there can be multiple values. If it has no use in the Ftree algorithm, is it worth spending time to set all the hop values ? Nicolas From kliteyn at dev.mellanox.co.il Thu Mar 12 05:16:32 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Thu, 12 Mar 2009 14:16:32 +0200 Subject: [ofa-general] [PATCH] opensm/osm_ucast_ftree.c: some refactoring In-Reply-To: <49B8FABB.3080402@ext.bull.net> References: <49B8D73D.6060009@dev.mellanox.co.il> <49B8FABB.3080402@ext.bull.net> Message-ID: <49B8FD20.9090304@dev.mellanox.co.il> Nicolas Morey Chaisemartin wrote: > Hi, > > Are you sure of this? > Seems to me as if there is a problem > Yevgeny Kliteynik wrote: > > ... > > Function is setting hops value for the origin ports of the given port group on the given switch >> /*************************************************** >> @@ -2027,19 +2040,11 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, >> >> /* On the remote switch that is pointed by the p_group, >> set hops for ALL the ports in the remote group. */ >> - >> - for (j = 0; j < ports_num; j++) { >> - cl_ptr_vector_at(&p_group->ports, j, >> - (void *)&p_port); >> - >> - sw_set_hops(p_remote_sw, >> - cl_ntoh16(target_lid), >> - p_port->remote_port_num, >> - ((target_rank - highest_rank_in_route) + >> - (p_remote_sw->rank - highest_rank_in_route) >> - + reverse_hops * 2)); >> - } >> - >> + sw_set_hops(p_remote_sw, >> + cl_ntoh16(target_lid), p_group, >> + ((target_rank - highest_rank_in_route) + >> + (p_remote_sw->rank - highest_rank_in_route) + >> + reverse_hops * 2)); >> } > > Here you call the function wth the remote switch and a local port group. > So if I'm write the port num you use on the function are the port num on p_sw and not p_remote_sw... Oh, you're right. Thanks. I'll fix it. > The remote port group should be pass to this function but I don't think we have an easy way to get it. > Other solution could be to factorize the code in another function which would call sw_set_hops and provide it the right port num. > > Anyway, how is the hops table used? > In the ftree algorithm, it seems we only use the min value through osm_switch_get_least_hops so we don't use the fact that there can be multiple values. > If it has no use in the Ftree algorithm, is it worth spending time to set all the hop values ? Hops tables are used for multicast routing, so if the matrix is not full, there will be more bottlenecks in multicast routes. -- Yevgeny > > Nicolas > From sashak at voltaire.com Thu Mar 12 05:28:06 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 14:28:06 +0200 Subject: [ofa-general] ibdmchk: How to generate PSL and SL2VL files ? In-Reply-To: <829ded920903112322r728b65d8h3a78b67141303b3b@mail.gmail.com> References: <829ded920903112322r728b65d8h3a78b67141303b3b@mail.gmail.com> Message-ID: <20090312122806.GB6766@sashak.voltaire.com> Hi Manesh, On 11:52 Thu 12 Mar , Keshetti Mahesh wrote: > > But I am not able to generate any such file with OpenSM. And there isn't any > information related to these files in man page as well as help(-h). Can anyone > please tell me how to generate path<->SL and SL2VL files using OpenSM? OpenSM doesn't generate such files. 'ibdmchk --help' states that OpenSM can be used for fdb and mcfdb files generation (not psl, slvl files). Sasha From sashak at voltaire.com Thu Mar 12 05:36:48 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 14:36:48 +0200 Subject: [ofa-general] [PATCH 11/10] libibmad:infiniband-diags: deprecate madrpc_set_[retries|timeout] WAS: [PATCH 1/10] libibmad: Clean up "new" interface In-Reply-To: <20090311090941.fde8d90c.weiny2@llnl.gov> References: <20090220143402.c3b23b0a.weiny2@llnl.gov> <20090301072622.GG3936@sashak.voltaire.com> <20090302111101.bc822db8.weiny2@llnl.gov> <20090307191647.GJ9673@sashak.voltaire.com> <20090310163426.63d67ff2.weiny2@llnl.gov> <20090311102941.GB22085@sashak.voltaire.com> <20090311090941.fde8d90c.weiny2@llnl.gov> Message-ID: <20090312123640.GC6766@sashak.voltaire.com> On 09:09 Wed 11 Mar , Ira Weiny wrote: > > OK! I miss-understood you. I thought you did not want to add another set of > timeout/retry values. I didn't, but you was against me and thought that per port timeout/retries can be useful. > The below is just fine with me. Do you ever see > getting rid of the global default? I would leave it in order to simplify initial setup for diag tools - IOW don't want to enforce application to call mad_rpc_set_timeout() explicitly, the default should work just fine. Sasha From hal.rosenstock at gmail.com Thu Mar 12 05:31:47 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 12 Mar 2009 08:31:47 -0400 Subject: ***SPAM*** Re: [ofa-general] can't use perfquery from git against late kernels In-Reply-To: <49B8FA1B.1030800@Voltaire.com> References: <49B8FA1B.1030800@Voltaire.com> Message-ID: On Thu, Mar 12, 2009 at 8:03 AM, Or Gerlitz wrote: > Hal Rosenstock wrote: >> Looks like QKey is wrong (no longer being set ?) in request MAD. > > Thanks for looking into that, grepping for qkey on both outputs doesn't show any difference > both use 0x80010000 ?! There were QKey traps at the SM indicating QKey 0. Sasha's rpc.c fix seems to take care of this. -- Hal > Or. > > # grep -i qkey perfquery-* > perfquery-git-latest-4ec6120d85c0ccfb4fb62cb8b50f27411c81e268:ibwarn: [8304] umad_set_addr: umad 0x7fff04a01bc0 dlid 4 dqp 1 sl 0, qkey 80010000 > perfquery-git-latest-4ec6120d85c0ccfb4fb62cb8b50f27411c81e268:ibwarn: [8304] umad_addr_dump: qpn 1 qkey 0x80010000 lid 4 sl 0 > perfquery-git-latest-4ec6120d85c0ccfb4fb62cb8b50f27411c81e268:ibwarn: [8304] umad_addr_dump: qpn 1 qkey 0x80010000 lid 4 sl 0 > perfquery-git-latest-4ec6120d85c0ccfb4fb62cb8b50f27411c81e268:ibwarn: [8304] umad_addr_dump: qpn 1 qkey 0x80010000 lid 4 sl 0 > perfquery-infiniband-diags-1.4.3_20081127_a92a5e5-1.ofed1.4.rc6:ibwarn: [8273] umad_set_addr: umad 0x7fffe8c9a620 dlid 4 dqp 1 sl 0, qkey 80010000 > perfquery-infiniband-diags-1.4.3_20081127_a92a5e5-1.ofed1.4.rc6:ibwarn: [8273] umad_addr_dump: qpn 1 qkey 0x80010000 lid 0x4 sl 0 > perfquery-infiniband-diags-1.4.3_20081127_a92a5e5-1.ofed1.4.rc6:ibwarn: [8273] umad_set_addr: umad 0x7fffe8c9a1d0 dlid 4 dqp 1 sl 0, qkey 80010000 > perfquery-infiniband-diags-1.4.3_20081127_a92a5e5-1.ofed1.4.rc6:ibwarn: [8273] umad_addr_dump: qpn 1 qkey 0x80010000 lid 0x4 sl 0 > From sashak at voltaire.com Thu Mar 12 06:12:48 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 15:12:48 +0200 Subject: [ofa-general] Re: [PATCH 1/2] include/opensm/osm_opensm.h add setup function to routing engine. In-Reply-To: <49A6B618.1090300@gmail.com> References: <49A6B618.1090300@gmail.com> Message-ID: <20090312131241.GA8818@sashak.voltaire.com> Hi Eli, On 17:32 Thu 26 Feb , Eli Dorfman (Voltaire) wrote: > add setup function to routing engine. You don't need to split patches like this. Both looks like a single logical change. Sasha From hal.rosenstock at gmail.com Thu Mar 12 06:16:08 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 12 Mar 2009 09:16:08 -0400 Subject: [ofa-general] [PATCH 11/10] libibmad:infiniband-diags: deprecate madrpc_set_[retries|timeout] WAS: [PATCH 1/10] libibmad: Clean up "new" interface In-Reply-To: <20090311102941.GB22085@sashak.voltaire.com> References: <20090219190525.322681b8.weiny2@llnl.gov> <20090220143402.c3b23b0a.weiny2@llnl.gov> <20090301072622.GG3936@sashak.voltaire.com> <20090302111101.bc822db8.weiny2@llnl.gov> <20090307191647.GJ9673@sashak.voltaire.com> <20090310163426.63d67ff2.weiny2@llnl.gov> <20090311102941.GB22085@sashak.voltaire.com> Message-ID: On Wed, Mar 11, 2009 at 6:29 AM, Sasha Khapyorsky wrote: > On 16:34 Tue 10 Mar     , Ira Weiny wrote: >> >> I will look at documenting the above as technically it is possible to change >> the timeout on various ports by setting the parameters appropriately on each >> call.  I don't like this going forward but I guess it is doable without >> changing the interface. > > And what do you say about the patch below? It doesn't change the current > diags behavior and provides per port timeout/retries setup. The global > parameters and rpc's fields remain to be same, and we don;t need to > deprecate madrpc_set_timeout/retires(). > > Sasha > > > >From 25060f5a7243a1bae6a7e7e9b2b79051ab0473d9 Mon Sep 17 00:00:00 2001 > From: Sasha Khapyorsky > Date: Wed, 11 Mar 2009 12:15:54 +0200 > Subject: [PATCH] libibmad: per port timeout and retires setup > > This adds functions mad_rpc_set_timeout() and mad_rpc_set_retries() > which provides possibility to specify timeout and retries for mad rpc > processing per port. This doesn't change the default behavior. > > Signed-off-by: Sasha Khapyorsky > --- >  libibmad/include/infiniband/mad.h |    2 ++ >  libibmad/src/libibmad.map         |    2 ++ >  libibmad/src/mad_internal.h       |    1 + >  libibmad/src/rpc.c                |   37 +++++++++++++++++++++++++++---------- >  4 files changed, 32 insertions(+), 10 deletions(-) > > diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h > index 064cbb7..9b398fa 100644 > --- a/libibmad/include/infiniband/mad.h > +++ b/libibmad/include/infiniband/mad.h > @@ -742,6 +742,8 @@ MAD_EXPORT void *mad_rpc(const struct ibmad_port *srcport, ib_rpc_t * rpc, ib_po >  MAD_EXPORT void *mad_rpc_rmpp(const struct ibmad_port *srcport, ib_rpc_t * rpc, ib_portid_t * dport, >                        ib_rmpp_hdr_t * rmpp, void *data); >  MAD_EXPORT int mad_rpc_portid(struct ibmad_port *srcport); > +MAD_EXPORT void mad_rpc_set_retries(struct ibmad_port *port, int retries); > +MAD_EXPORT void mad_rpc_set_timeout(struct ibmad_port *port, int timeout); > >  /* register.c */ >  MAD_EXPORT int mad_register_port_client(int port_id, int mgmt, > diff --git a/libibmad/src/libibmad.map b/libibmad/src/libibmad.map > index 0be7a92..10c7cd0 100644 > --- a/libibmad/src/libibmad.map > +++ b/libibmad/src/libibmad.map > @@ -68,6 +68,8 @@ IBMAD_1.3 { >                mad_rpc; >                mad_rpc_rmpp; >                mad_rpc_portid; > +               mad_rpc_set_retries; > +               mad_rpc_set_timeout; >                madrpc; >                madrpc_def_timeout; >                madrpc_init; > diff --git a/libibmad/src/mad_internal.h b/libibmad/src/mad_internal.h > index e948540..24418cc 100644 > --- a/libibmad/src/mad_internal.h > +++ b/libibmad/src/mad_internal.h > @@ -39,6 +39,7 @@ >  struct ibmad_port { >        int port_id;            /* file descriptor returned by umad_open() */ >        int class_agents[MAX_CLASS];    /* class2agent mapper */ > +       int timeout, retries; >  }; > >  extern struct ibmad_port *ibmp; > diff --git a/libibmad/src/rpc.c b/libibmad/src/rpc.c > index 8c68cf9..3f3b7a2 100644 > --- a/libibmad/src/rpc.c > +++ b/libibmad/src/rpc.c > @@ -53,7 +53,7 @@ struct ibmad_port *ibmp = &mad_port; >  static int iberrs; > >  static int madrpc_retries = MAD_DEF_RETRIES; > -static int def_madrpc_timeout = MAD_DEF_TIMEOUT_MS; > +static int madrpc_timeout = MAD_DEF_TIMEOUT_MS; >  static void *save_mad; >  static int save_mad_len = 256; > > @@ -83,13 +83,23 @@ int madrpc_set_retries(int retries) > >  int madrpc_set_timeout(int timeout) >  { > -       def_madrpc_timeout = timeout; > +       madrpc_timeout = timeout; >        return 0; >  } > > +void mad_rpc_set_retries(struct ibmad_port *port, int retries) > +{ > +       port->retries = retries; > +} > + > +void mad_rpc_set_timeout(struct ibmad_port *port, int timeout) > +{ > +       port->timeout = timeout; > +} > + >  int madrpc_def_timeout(void) >  { > -       return def_madrpc_timeout; > +       return madrpc_timeout; >  } > >  int madrpc_portid(void) > @@ -104,15 +114,12 @@ int mad_rpc_portid(struct ibmad_port *srcport) > >  static int >  _do_madrpc(int port_id, void *sndbuf, void *rcvbuf, int agentid, int len, > -          int timeout) > +          int timeout, int max_retries) >  { >        uint32_t trid;          /* only low 32 bits */ >        int retries; >        int length, status; > > -       if (!timeout) > -               timeout = def_madrpc_timeout; > - >        if (ibdebug > 1) { >                IBWARN(">>> sending: len %d pktsz %zu", len, umad_size() + len); >                xdump(stderr, "send buf\n", sndbuf, umad_size() + len); > @@ -127,7 +134,7 @@ _do_madrpc(int port_id, void *sndbuf, void *rcvbuf, int agentid, int len, >        trid = >            (uint32_t) mad_get_field64(umad_get_mad(sndbuf), 0, IB_MAD_TRID_F); > > -       for (retries = 0; retries < madrpc_retries; retries++) { > +       for (retries = 0; retries < max_retries; retries++) { >                if (retries) { >                        ERRS("retry %d (timeout %d ms)", retries, timeout); >                } > @@ -171,6 +178,7 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport >  { >        int status, len; >        uint8_t sndbuf[1024], rcvbuf[1024], *mad; > +       int timeout, retries; > >        len = 0; >        memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); > @@ -178,9 +186,13 @@ void *mad_rpc(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * dport >        if ((len = mad_build_pkt(sndbuf, rpc, dport, 0, payload)) < 0) >                return 0; > > +       timeout = rpc->timeout ? rpc->timeout : > +                       port->timeout ? port->timeout : madrpc_timeout; > +       retries = port->retries ? port->retries : madrpc_retries; > + >        if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, >                              port->class_agents[rpc->mgtclass], > -                             len, rpc->timeout)) < 0) { > +                             len, timeout, retries)) < 0) { >                IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); >                return 0; >        } > @@ -209,6 +221,7 @@ void *mad_rpc_rmpp(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * >  { >        int status, len; >        uint8_t sndbuf[1024], rcvbuf[1024], *mad; > +       int timeout, retries; > >        memset(sndbuf, 0, umad_size() + IB_MAD_SIZE); > > @@ -217,9 +230,13 @@ void *mad_rpc_rmpp(const struct ibmad_port *port, ib_rpc_t * rpc, ib_portid_t * >        if ((len = mad_build_pkt(sndbuf, rpc, dport, rmpp, data)) < 0) >                return 0; > > +       timeout = rpc->timeout ? rpc->timeout : > +                       port->timeout ? port->timeout : madrpc_timeout; > +       retries = port->retries ? port->retries : madrpc_retries; > + >        if ((len = _do_madrpc(port->port_id, sndbuf, rcvbuf, >                              port->class_agents[rpc->mgtclass], > -                             len, rpc->timeout)) < 0) { > +                             len, rpc->timeout, retries)) < 0) { Shouldn't this parameter be timeout r.t. rpc->timeout ? -- Hal >                IBWARN("_do_madrpc failed; dport (%s)", portid2str(dport)); >                return 0; >        } > -- > 1.6.1.2.319.gbd9e > > From sashak at voltaire.com Thu Mar 12 06:21:44 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 15:21:44 +0200 Subject: [ofa-general] Re: [PATCH 2/2] opensm: setup routing engine when in use and delete when fail In-Reply-To: <49A6B6EB.80700@gmail.com> References: <49A6B618.1090300@gmail.com> <49A6B6EB.80700@gmail.com> Message-ID: <20090312132137.GB8818@sashak.voltaire.com> On 17:36 Thu 26 Feb , Eli Dorfman (Voltaire) wrote: > setup routing engine when in use and delete when fail > setup routing engine before use. > delete resources when routing algorithm fails > this will save allocation for routing algorithms that are not used. > > Signed-off-by: Eli Dorfman > --- > opensm/opensm/osm_opensm.c | 20 ++++++-------------- > opensm/opensm/osm_ucast_mgr.c | 34 +++++++++++++++++++++++++++++++++- > 2 files changed, 39 insertions(+), 15 deletions(-) > > diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c > index 7de2e5b..a2620d5 100644 > --- a/opensm/opensm/osm_opensm.c > +++ b/opensm/opensm/osm_opensm.c > @@ -169,21 +169,14 @@ static void setup_routing_engine(osm_opensm_t *osm, const char *name) > memset(re, 0, sizeof(struct osm_routing_engine)); > > re->name = m->name; > - if (m->setup(re, osm)) { > - OSM_LOG(&osm->log, OSM_LOG_VERBOSE, > - "setup of routing" > - " engine \'%s\' failed\n", name); > - return; > - } > - OSM_LOG(&osm->log, OSM_LOG_DEBUG, > - "\'%s\' routing engine set up\n", re->name); > + re->setup = m->setup; > append_routing_engine(osm, re); > return; > } > } > > OSM_LOG(&osm->log, OSM_LOG_ERROR, > - "cannot find or setup routing engine \'%s\'", name); > + "cannot find or setup routing engine \'%s\'\n", name); This is needed fix, but unrelated to the patch. > } > > static void setup_routing_engines(osm_opensm_t *osm, const char *engine_names) > @@ -224,18 +217,17 @@ void osm_opensm_construct(IN osm_opensm_t * const p_osm) > > /********************************************************************** > **********************************************************************/ > -static void destroy_routing_engines(osm_opensm_t *osm) > +static void destroy_routing_engines(struct osm_routing_engine **re) Any reason for function prototype change? > { > struct osm_routing_engine *r, *next; > > - next = osm->routing_engine_list; > + next = *re; > while (next) { > r = next; > next = r->next; > - if (r->delete) > - r->delete(r->context); What will happen with currently used routing engine? It will be never deleted, leak? right? > free(r); > } > + *re = NULL; > } > > /********************************************************************** > @@ -289,7 +281,7 @@ void osm_opensm_destroy(IN osm_opensm_t * const p_osm) > > /* do the destruction in reverse order as init */ > destroy_plugins(p_osm); > - destroy_routing_engines(p_osm); > + destroy_routing_engines(&p_osm->routing_engine_list); > osm_sa_destroy(&p_osm->sa); > osm_sm_destroy(&p_osm->sm); > #ifdef ENABLE_OSM_PERF_MGR > diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c > index e404c91..7175926 100644 > --- a/opensm/opensm/osm_ucast_mgr.c > +++ b/opensm/opensm/osm_ucast_mgr.c > @@ -886,7 +886,6 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) > > p_sw_guid_tbl = &p_mgr->p_subn->sw_guid_tbl; > p_osm = p_mgr->p_subn->p_osm; > - p_routing_eng = p_osm->routing_engine_list; > > CL_PLOCK_EXCL_ACQUIRE(p_mgr->p_lock); > > @@ -897,10 +896,30 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) > ucast_mgr_setup_all_switches(p_mgr->p_subn) < 0) > goto Exit; > > + /* update the entry in active list */ > + > p_osm->routing_engine_used = OSM_ROUTING_ENGINE_TYPE_NONE; > + p_routing_eng = p_osm->routing_engine_list; > while (p_routing_eng) { > + if (!p_routing_eng->initialized && > + p_routing_eng->setup(p_routing_eng, p_osm)) { ^^^^^^^^ > + OSM_LOG(p_mgr->p_log, OSM_LOG_VERBOSE, > + "setup of routing engine \'%s\' failed\n", > + p_routing_eng->name); ^^^^^^^^ Keep indentation please. > + p_routing_eng = p_routing_eng->next; > + continue; > + } > + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, > + "\'%s\' routing engine set up\n", p_routing_eng->name); > + p_routing_eng->initialized = 1; > + > if (!ucast_mgr_route(p_routing_eng, p_osm)) > break; > + > + /* delete unused routing engine */ > + if (p_routing_eng->delete) > + p_routing_eng->delete(p_routing_eng->context); > + > p_routing_eng = p_routing_eng->next; > } > > @@ -911,6 +930,19 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) > p_osm->routing_engine_used = OSM_ROUTING_ENGINE_TYPE_MINHOP; > } > > + /* if for some reason different routing engine is used */ > + /* cleanup unused routing engine */ > + p_routing_eng = p_osm->routing_engine_list; > + while (p_routing_eng) { > + if (p_routing_eng->initialized && > + p_osm->routing_engine_used != > + osm_routing_engine_type(p_routing_eng->name) && > + p_routing_eng->delete) > + p_routing_eng->delete(p_routing_eng->context); > + > + p_routing_eng = p_routing_eng->next; > + } > + How this section of code is useful? Sasha From sashak at voltaire.com Thu Mar 12 06:24:51 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 15:24:51 +0200 Subject: [ofa-general] [PATCH 11/10] libibmad:infiniband-diags: deprecate madrpc_set_[retries|timeout] WAS: [PATCH 1/10] libibmad: Clean up "new" interface In-Reply-To: References: <20090220143402.c3b23b0a.weiny2@llnl.gov> <20090301072622.GG3936@sashak.voltaire.com> <20090302111101.bc822db8.weiny2@llnl.gov> <20090307191647.GJ9673@sashak.voltaire.com> <20090310163426.63d67ff2.weiny2@llnl.gov> <20090311102941.GB22085@sashak.voltaire.com> Message-ID: <20090312132451.GC8818@sashak.voltaire.com> On 09:16 Thu 12 Mar , Hal Rosenstock wrote: > > - ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? len, rpc->timeout)) < 0) { > > + ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? len, rpc->timeout, retries)) < 0) { > > Shouldn't this parameter be timeout r.t. rpc->timeout ? Yes, it should. Thanks for finding this. Sasha From ogerlitz at Voltaire.com Thu Mar 12 06:20:27 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Thu, 12 Mar 2009 15:20:27 +0200 Subject: [ofa-general] Re: [PATCH] libibmad/rpc: fix class registration bug [was: Re: can't use perfquery from git against late kernels] In-Reply-To: <20090312121004.GA6766@sashak.voltaire.com> References: <20090312121004.GA6766@sashak.voltaire.com> Message-ID: <49B90C1B.4070808@Voltaire.com> Sasha Khapyorsky wrote: > This commit breaks tools which use non-SM class (such as perfquery). > There is the fix, I will commit soon. Hi Sasha, It fixed the problem - many thanks to you and Hal for the quick resolution Or. From sashak at voltaire.com Thu Mar 12 06:44:13 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 15:44:13 +0200 Subject: [ofa-general] Re: [PATCH 2/2] opensm: setup routing engine when in use and delete when fail In-Reply-To: <49A6B6EB.80700@gmail.com> References: <49A6B618.1090300@gmail.com> <49A6B6EB.80700@gmail.com> Message-ID: <20090312134406.GE8818@sashak.voltaire.com> On 17:36 Thu 26 Feb , Eli Dorfman (Voltaire) wrote: > diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c > index 7de2e5b..a2620d5 100644 > --- a/opensm/opensm/osm_opensm.c > +++ b/opensm/opensm/osm_opensm.c > @@ -169,21 +169,14 @@ static void setup_routing_engine(osm_opensm_t *osm, const char *name) > memset(re, 0, sizeof(struct osm_routing_engine)); > > re->name = m->name; > - if (m->setup(re, osm)) { > - OSM_LOG(&osm->log, OSM_LOG_VERBOSE, > - "setup of routing" > - " engine \'%s\' failed\n", name); > - return; > - } > - OSM_LOG(&osm->log, OSM_LOG_DEBUG, > - "\'%s\' routing engine set up\n", re->name); > + re->setup = m->setup; Also this make compile warning: osm_opensm.c: In function 'setup_routing_engine': osm_opensm.c:172: warning: assignment from incompatible pointer type Sasha From sashak at voltaire.com Thu Mar 12 06:47:05 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 15:47:05 +0200 Subject: [ofa-general] Re: [PATCH 1/2] include/opensm/osm_opensm.h support routing engine update In-Reply-To: <49A6B8A3.2020703@gmail.com> References: <49A6B8A3.2020703@gmail.com> Message-ID: <20090312134658.GF8818@sashak.voltaire.com> On 17:43 Thu 26 Feb , Eli Dorfman (Voltaire) wrote: > support routing engine update. > add prev routing engine list. Same as with previous series - you don't need to add function prototype in one patch and its implementation in another. > @@ -525,5 +526,7 @@ extern volatile unsigned int osm_exit_flag; > * Set to one to cause all threads to leave > *********/ > > +void update_routing_engines(osm_opensm_t *osm, const char *engine_names); > + For global names we are using 'osm_' prefix. Should be: void osm_update_routing_engines(osm_opensm_t *osm, const char *engine_names); Sasha From keshetti.mahesh at gmail.com Thu Mar 12 06:56:20 2009 From: keshetti.mahesh at gmail.com (Keshetti Mahesh) Date: Thu, 12 Mar 2009 19:26:20 +0530 Subject: ***SPAM*** Re: [ofa-general] ibdmchk: How to generate PSL and SL2VL files ? In-Reply-To: <20090312122806.GB6766@sashak.voltaire.com> References: <829ded920903112322r728b65d8h3a78b67141303b3b@mail.gmail.com> <20090312122806.GB6766@sashak.voltaire.com> Message-ID: <829ded920903120656i6eefce39g9dde2e16b4cdb1@mail.gmail.com> Thanks Sasha for your reply. > OpenSM doesn't generate such files. > > 'ibdmchk --help' states that OpenSM can be used for fdb and mcfdb files > generation (not psl, slvl files). Can you please tell me is there any other way(a script or plug in) to generate these files ? -Mahesh From sashak at voltaire.com Thu Mar 12 07:03:36 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 16:03:36 +0200 Subject: [ofa-general] Re: [PATCH 2/2] opensm routing engine update In-Reply-To: <49A6B9EE.7000008@gmail.com> References: <49A6B8A3.2020703@gmail.com> <49A6B9EE.7000008@gmail.com> Message-ID: <20090312140336.GG8818@sashak.voltaire.com> On 17:49 Thu 26 Feb , Eli Dorfman (Voltaire) wrote: > > diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c > index a2620d5..6ab28be 100644 > --- a/opensm/opensm/osm_opensm.c > +++ b/opensm/opensm/osm_opensm.c > @@ -230,6 +230,15 @@ static void destroy_routing_engines(struct osm_routing_engine **re) > *re = NULL; > } > > +void update_routing_engines(osm_opensm_t *osm, const char *engine_names) > +{ > + /* cleanup prev routing engine list and replace with current list */ > + destroy_routing_engines(&osm->prev_routing_engine_list); > + osm->prev_routing_engine_list = osm->routing_engine_list; > + osm->routing_engine_list = NULL; > + setup_routing_engines(osm, engine_names); > +} > + If you are going to setup/delete routing engine right before/after use (previous patch series) why do you need to keep whole list as previous? Could you just drop uninitialized REs? > diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c > index b3100a4..1ba5c91 100644 > --- a/opensm/opensm/osm_subnet.c > +++ b/opensm/opensm/osm_subnet.c > @@ -151,6 +151,14 @@ static void opts_setup_sm_priority(osm_subn_t *p_subn, void *p_val) > osm_set_sm_priority(p_sm, sm_priority); > } > > +static void opts_setup_routing_engine(osm_subn_t *p_subn, void *p_val) > +{ > + osm_opensm_t *p_osm = p_subn->p_osm; > + char *engines = (char *) p_val; Casting is not needed. Actually you don't need any intermediate variables here. Just something like this: osm_update_routing_engines(p_subn->p_osm, p_val); should work fine. > + > + update_routing_engines(p_osm, engines); > +} > + > static void opts_parse_net64(IN osm_subn_t *p_subn, IN char *p_key, > IN char *p_val_str, void *p_v1, void *p_v2, > void (*pfn)(osm_subn_t *, void *)) > diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c > index 7175926..cda9f34 100644 > --- a/opensm/opensm/osm_ucast_mgr.c > +++ b/opensm/opensm/osm_ucast_mgr.c > @@ -879,7 +879,7 @@ static int ucast_mgr_route(struct osm_routing_engine *r, osm_opensm_t *osm) > int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) > { > osm_opensm_t *p_osm; > - struct osm_routing_engine *p_routing_eng; > + struct osm_routing_engine *p_routing_eng, *r; > cl_qmap_t *p_sw_guid_tbl; > > OSM_LOG_ENTER(p_mgr->p_log); > @@ -896,6 +896,26 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) > ucast_mgr_setup_all_switches(p_mgr->p_subn) < 0) > goto Exit; > > + /* find used routing engine in previous list */ > + r = p_osm->prev_routing_engine_list; > + while (r) { > + if (p_osm->routing_engine_used == > + osm_routing_engine_type(r->name)) > + { > + p_routing_eng = p_osm->routing_engine_list; > + while (p_routing_eng) { > + if (p_osm->routing_engine_used == > + osm_routing_engine_type(p_routing_eng->name)) { > + memcpy(p_routing_eng, r, sizeof(*p_routing_eng)); > + break; > + } > + p_routing_eng = p_routing_eng->next; > + } > + break; > + } > + r = r->next; > + } > + I think that I'm able to understand what you are trying to achieve here. But could this be done less tricky (I'm fine to have more complicated code in osm_update_routing_engine() function instead)? For example osm_update_routing_engine() could save currently used RE as to_be_deleted_re field (in case when it was deleted) and drop it at end of the sweep. Sasha From sashak at voltaire.com Thu Mar 12 07:04:35 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 16:04:35 +0200 Subject: [ofa-general] ibdmchk: How to generate PSL and SL2VL files ? In-Reply-To: <829ded920903120656i6eefce39g9dde2e16b4cdb1@mail.gmail.com> References: <829ded920903112322r728b65d8h3a78b67141303b3b@mail.gmail.com> <20090312122806.GB6766@sashak.voltaire.com> <829ded920903120656i6eefce39g9dde2e16b4cdb1@mail.gmail.com> Message-ID: <20090312140435.GH8818@sashak.voltaire.com> On 19:26 Thu 12 Mar , Keshetti Mahesh wrote: > > Can you please tell me is there any other way(a script or plug in) to generate > these files ? I have no idea, sorry. Sasha From hnrose at comcast.net Thu Mar 12 07:57:32 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 12 Mar 2009 09:57:32 -0500 Subject: [ofa-general] ***SPAM*** [PATCH][TRIVIAL] libibmad/register.c: Cosmetic formatting change Message-ID: <20090312145732.GA18900@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/libibmad/src/register.c b/libibmad/src/register.c index eb8a62e..eac8f5d 100644 --- a/libibmad/src/register.c +++ b/libibmad/src/register.c @@ -90,9 +90,8 @@ int mad_register_port_client(int port_id, int mgmt, uint8_t rmpp_version) } agent = umad_register(port_id, mgmt, vers, rmpp_version, 0); - if (agent < 0) { + if (agent < 0) DEBUG("Can't register agent for class %d", mgmt); - } return agent; } From dorfman.eli at gmail.com Thu Mar 12 07:15:10 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Thu, 12 Mar 2009 16:15:10 +0200 Subject: [ofa-general] [PATCH v2] opensm: set IS_SM bit during opensm init In-Reply-To: <49AFC929.20606@gmail.com> References: <49AFC929.20606@gmail.com> Message-ID: <49B918EE.5010504@gmail.com> Set IS_SM bit during opensm init Set IS_SM bit during opensm initalization. This fixes a problem of SM that enters to standby and has mismatch between local capability (IS_SM off) and the SMA. Signed-off-by: Eli Dorfman --- opensm/opensm/osm_opensm.c | 5 +++++ opensm/opensm/osm_port_info_rcv.c | 35 +---------------------------------- 2 files changed, 6 insertions(+), 34 deletions(-) diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c index 7de2e5b..cfe6474 100644 --- a/opensm/opensm/osm_opensm.c +++ b/opensm/opensm/osm_opensm.c @@ -475,6 +475,11 @@ osm_opensm_bind(IN osm_opensm_t * const p_osm, IN const ib_net64_t guid) goto Exit; #endif /* ENABLE_OSM_PERF_MGR */ + /* setting IS_SM in capability mask */ + OSM_LOG(&p_osm->log, OSM_LOG_INFO, "Setting IS_SM on port 0x%016" PRIx64 "\n", + cl_ntoh64(guid)); + osm_vendor_set_sm(p_osm->sm.mad_ctrl.h_bind, TRUE); + Exit: OSM_LOG_EXIT(&p_osm->log); return (status); diff --git a/opensm/opensm/osm_port_info_rcv.c b/opensm/opensm/osm_port_info_rcv.c index 3e39dff..7a3e487 100644 --- a/opensm/opensm/osm_port_info_rcv.c +++ b/opensm/opensm/osm_port_info_rcv.c @@ -62,30 +62,6 @@ #include #include -/********************************************************************** - **********************************************************************/ -static void -__osm_pi_rcv_set_sm(IN osm_sm_t * sm, - IN osm_physp_t * const p_physp) -{ - osm_bind_handle_t h_bind; - osm_dr_path_t *p_dr_path; - - OSM_LOG_ENTER(sm->p_log); - - OSM_LOG(sm->p_log, OSM_LOG_DEBUG, - "Setting IS_SM bit in port attributes\n"); - - p_dr_path = osm_physp_get_dr_path_ptr(p_physp); - h_bind = osm_dr_path_get_bind_handle(p_dr_path); - - /* - The 'IS_SM' bit isn't already set, so set it. - */ - osm_vendor_set_sm(h_bind, TRUE); - - OSM_LOG_EXIT(sm->p_log); -} /********************************************************************** **********************************************************************/ @@ -139,16 +115,7 @@ __osm_pi_rcv_process_endport(IN osm_sm_t * sm, } } - if (port_guid == sm->p_subn->sm_port_guid) { - /* - We received the PortInfo for our own port. - */ - if (!(p_pi->capability_mask & IB_PORT_CAP_IS_SM)) - /* - Set the IS_SM bit to indicate our port hosts an SM. - */ - __osm_pi_rcv_set_sm(sm, p_physp); - } else { + if (port_guid != sm->p_subn->sm_port_guid) { p_sm_tbl = &sm->p_subn->sm_guid_tbl; if (p_pi->capability_mask & IB_PORT_CAP_IS_SM) { /* -- 1.5.5 From sashak at voltaire.com Thu Mar 12 07:14:52 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 16:14:52 +0200 Subject: [ofa-general] Re: [PATCH][TRIVIAL] libibmad/register.c: Cosmetic formatting change In-Reply-To: <20090312145732.GA18900@comcast.net> References: <20090312145732.GA18900@comcast.net> Message-ID: <20090312141452.GK8818@sashak.voltaire.com> On 09:57 Thu 12 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Thu Mar 12 07:17:47 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 16:17:47 +0200 Subject: [ofa-general] Re: [PATCH] opensm/osm_subnet.c: fixing compiler warnings In-Reply-To: <49B8F9D8.9090506@dev.mellanox.co.il> References: <49B8F9D8.9090506@dev.mellanox.co.il> Message-ID: <20090312141747.GL8818@sashak.voltaire.com> On 14:02 Thu 12 Mar , Yevgeny Kliteynik wrote: > Hi Sasha, > > There are couple of meaningless CL_ASSERT() lines in the > parsing functions that are checking the parsed value. > The compiler says that "comparison is always true due > to limited range of data type". So either the 'val' type > should be more general when parsing it, or the CL_ASSERT() > lines are not needed. The following patch removes them. > > Signed-off-by: Yevgeny Kliteynik Applied. Thanks. Sasha From sashak at voltaire.com Thu Mar 12 07:31:15 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 16:31:15 +0200 Subject: [ofa-general] Re: [PATCH] infiniband-diags/perfquery.8: Update man page for PortXmit/RcvDataSL In-Reply-To: <20090310205023.GA1791@comcast.net> References: <20090310205023.GA1791@comcast.net> Message-ID: <20090312143115.GM8818@sashak.voltaire.com> On 15:50 Tue 10 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock This patch does not apply against main stream, I fixed it by hand. Please patches you post against master. Applied. Thanks. Sasha From sashak at voltaire.com Thu Mar 12 07:31:32 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 16:31:32 +0200 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_trap_rcv.c: Remove extraneous comment In-Reply-To: <20090310192425.GC6431@comcast.net> References: <20090310192425.GC6431@comcast.net> Message-ID: <20090312143132.GN8818@sashak.voltaire.com> On 14:24 Tue 10 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Thu Mar 12 07:42:07 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 16:42:07 +0200 Subject: [ofa-general] Re: [PATCH][TRIVIAL] libibmad/mad.h: Cosmetic formatting changes In-Reply-To: <20090310225114.GB3880@comcast.net> References: <20090310225114.GB3880@comcast.net> Message-ID: <20090312144207.GP8818@sashak.voltaire.com> On 17:51 Tue 10 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Thu Mar 12 07:43:48 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 16:43:48 +0200 Subject: [ofa-general] Re: [PATCH] infiniband-diags/vendstat.c: Add missing mad_rpc_close_port call In-Reply-To: <20090311132919.GA23957@comcast.net> References: <20090311132919.GA23957@comcast.net> Message-ID: <20090312144348.GQ8818@sashak.voltaire.com> On 08:29 Wed 11 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Thu Mar 12 07:54:59 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 16:54:59 +0200 Subject: [ofa-general] Re: [PATCH] infiniband-diags/mcm_rereg_test.c: Add missing mad_rpc_close_port call In-Reply-To: <20090311135320.GA29870@comcast.net> References: <20090311135320.GA29870@comcast.net> Message-ID: <20090312145459.GR8818@sashak.voltaire.com> On 08:53 Wed 11 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock > > --- > diff --git a/infiniband-diags/src/mcm_rereg_test.c b/infiniband-diags/src/mcm_rereg_test.c > index de5ea90..d087e4a 100644 > --- a/infiniband-diags/src/mcm_rereg_test.c > +++ b/infiniband-diags/src/mcm_rereg_test.c > @@ -492,9 +492,13 @@ int main(int argc, char **argv) > mad = umad_get_mad(umad); > > free(umad); > +#if 1 > + mad_rpc_close_port(srcport); > +#else > umad_unregister(port, agent); > umad_close_port(port); > umad_done(); > +#endif This is not so correct - SA class is registered directly by using umad_register() and not with rpc. So it rather should be: umad_unregister(port, agent); mad_rpc_close_port(srcport); OTOH libibmad's rpc is not used in this program (only mad formatting stuff). So I would recommend to remove mad_rpc_* things completely. Sasha From swise at opengridcomputing.com Thu Mar 12 07:59:23 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Thu, 12 Mar 2009 09:59:23 -0500 Subject: [ofa-general] RE: tcp/rdma port unification patch in librdmacm In-Reply-To: <49B8B55C.50105@voltaire.com> References: <49B17E15.4050406@opengridcomputing.com> <811553D6D34A4E6F9AB686CA2E441D35@amr.corp.intel.com> <49B61C5D.6060900@Voltaire.com> <3281521B4F4A415A846A9F2E8AF9C21F@amr.corp.intel.com> <49B767DB.2050700@voltaire.com> <49B7D41C.4030707@opengridcomputing.com> <49B8B55C.50105@voltaire.com> Message-ID: <49B9234B.7070805@opengridcomputing.com> Or Gerlitz wrote: > Steve Wise wrote: >> Or maybe map PS_TCP port space to the be the unified space, and have >> a new PS_RDMA or PS_IB port space that doesn't? At and second > Please note that the rdma-cm derives the IB Service-ID from the port > space and the port provided to rdma_resolve_addr - and this is done > along the lines of the IBTA "RDMA IP CM Service" annex A11. > So we would need to have PS_IB_UD and PS_IB_RC that SID(ps=TCP, > port=X) = SID(ps=IB_RC, port=X) > at first thought this is possible and would add too much compatibility > issues - except for the need to provide an app with the means to sense > if the underlying librdmacm support the _IB port spaces. > > Still, I would like to avoid the case where someone runs MPI job over > RDMA-CM over IB and librdmacm opens/bounds to all these sockets - so > the preferred solution - for my taste would be some setting done by > the user to have librdmacm allocate socket / carve a port - and it > wouldn't be the default. For example, using an environment variable, > e.g similar to the libibverbs fork support. > I could live with that. It would allow taking advantage of this functionality without changing all the apps. From swise at opengridcomputing.com Thu Mar 12 08:04:24 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Thu, 12 Mar 2009 10:04:24 -0500 Subject: [ofa-general] Re: tcp/rdma port unification - iscsi/ddp/toe vs iwarp/ddp/toe In-Reply-To: <49B8BF05.5030004@Voltaire.com> References: <49B17E15.4050406@opengridcomputing.com> <49B76CD5.705@voltaire.com> <49B7D3A0.5060307@opengridcomputing.com> <15ddcffd0903110819s3b83cec0vb87ee6114496858e@mail.gmail.com> <49B7D899.1030601@opengridcomputing.com> <49B8BF05.5030004@Voltaire.com> Message-ID: <49B92478.4050604@opengridcomputing.com> Or Gerlitz wrote: > Steve Wise wrote: > >> Depends on the 4-tuple. The demux done in the TOE hw uses src addr/port and dst addr/port. >> > > Hi Steve, > > Trying to better understand the problem and possible solutions, I looked on the Chelsio iscsi offload driver (drivers/scsi/cxgb3i/* & Documentation/scsi/cxgb3i.txt) - which basically should suffer from the same issue - how to distinguish between a iscsi/os-tcp connection to iscsi/hw-tcp (toe) connection (or to iperf connection, etc as we discussed). > > I saw it manages the source port from its own data-structure (similar to the rdma-cm) but it doesn't open of a socket / nor bind the port - so I assume that a host opens two iscsi/tcp connections to a target on the same src ip/port - a party starts... At this point I came a cross the "cxgb3: manage private iSCSI IP address" patch whose change log states that its mostly ARP related- is the ARP resolution for iSCSI/DDP/TOE different from iWARP/DDP/TOE? does the below patch/approach eliminates the port space problem for iscsi/ddp? > > The iscsi offload drivers all maintain their own private ip addresses hidden from the native stack. That's how they keep the 4-tuple unique. Steve. From sashak at voltaire.com Thu Mar 12 08:18:10 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 17:18:10 +0200 Subject: [ofa-general] [PATCH v2] fix local port smlid In-Reply-To: <49AFE1E7.3030803@gmail.com> References: <49AFCCC0.6090507@gmail.com> <49AFE1E7.3030803@gmail.com> Message-ID: <20090312151810.GS8818@sashak.voltaire.com> On 16:29 Thu 05 Mar , Eli Dorfman (Voltaire) wrote: > fix local port smlid > > when opensm goes from master to standby due to handover its local > port smlid will be wrong (equal to its own lid). > the new master opensm will change local smlid only after local sm > (previous master) is in standby and will not query its port again. > Update our smlid with master opensm base lid > > Signed-off-by: Eli Dorfman > --- When posting V(N) of the patch, could you please describe under '---' line what was changed against V(N-1). Otherwise I need to apply both to some different branches and compare. Sasha From ogerlitz at Voltaire.com Thu Mar 12 08:13:08 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Thu, 12 Mar 2009 17:13:08 +0200 Subject: [ofa-general] Re: tcp/rdma port unification - iscsi/ddp/toe vs iwarp/ddp/toe In-Reply-To: <49B92478.4050604@opengridcomputing.com> References: <49B17E15.4050406@opengridcomputing.com> <49B76CD5.705@voltaire.com> <49B7D3A0.5060307@opengridcomputing.com> <15ddcffd0903110819s3b83cec0vb87ee6114496858e@mail.gmail.com> <49B7D899.1030601@opengridcomputing.com> <49B8BF05.5030004@Voltaire.com> <49B92478.4050604@opengridcomputing.com> Message-ID: <49B92684.4070902@Voltaire.com> Steve Wise wrote: > The iscsi offload drivers all maintain their own private ip addresses > hidden from the native stack. That's how they keep the 4-tuple unique. OK, so if the ip address wasn't private then the iscsi offload arp flow would have been the same as the iwarp one? and also the other way around, an iwarp stack can achieve the same impact (unique 4-tuple) either with shared port space or with private ip address - and actually, this ip address need not be private - it can be just --different-- from the one used for OS TCP. Anyway, bottom line, I see that you and Karen have chosen different approaches for the solution of the same problem (over the same HW...) and special reason for that - or it was a matter of how things were evolving? Or. From sashak at voltaire.com Thu Mar 12 08:21:04 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 17:21:04 +0200 Subject: ***SPAM*** Re: [ofa-general] [PATCH v2] fix local port smlid In-Reply-To: References: <49AFCCC0.6090507@gmail.com> <49AFE1E7.3030803@gmail.com> Message-ID: <20090312152057.GT8818@sashak.voltaire.com> On 09:33 Thu 05 Mar , Hal Rosenstock wrote: > > -- > > 1.5.5 > > Looks to me like the patch wasn't updated. It was: diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c index aa4609c..33196d1 100644 --- a/opensm/opensm/osm_req.c +++ b/opensm/opensm/osm_req.c @@ -243,7 +243,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) /* wrong value in local portinfo */ /* this may happen when we move to standby and master updates */ /* out smlid but we will not query it */ - pi->master_sm_base_lid = smport->p_physp->port_info.master_sm_base_lid; + pi->master_sm_base_lid = smport->p_physp->port_info.base_lid; /* don't bother with sending trap when SMA supports this */ if (!local && Sasha From tziporet at mellanox.co.il Thu Mar 12 08:15:58 2009 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Thu, 12 Mar 2009 17:15:58 +0200 Subject: [ofa-general] OFED meeting agenda for today Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD02080B53@mtlexch01.mtl.com> This is OFED meeting agenda for today (March 12): a. OFED 1.4.1 release status: Reminder for OFED 1.4.1 schedule: * RC1 - Mar 4 - done * RC2 - Mar 17 * RC3 - Mar 31 * GA - Apr 7 Critical bugs: bug_id bug_severity op_sys assigned_to short_short_desc 1538 blocker Other bugzilla at openib.org how to deal with /usr/src/ofa_kernel/include/linux/autoconf.h? 1537 blocker RHEL 5 jsquyres at cisco.com openmpi_gcc compilation failure 1525 blocker RHEL 5 vlad at mellanox.co.il OFED 1.4.1 fails to compile of EL5.3 on IA64 1546 blocker Other vlad at mellanox.co.il Compilation on SLES11 (x86_64 archi') failed 1540 critical SLES 10 andy.grover at oracle.com openib stop hangs on sles10sp2, ppc64 after kernel Oops in rds 1524 critical RHEL 5 eli at mellanox.co.il Running out of MTT segments, Intel server - 24 cores/64GB memory 1547 critical Other eli at mellanox.co.il modules are not being loaded after openibd restart on SLES11 1548 critical Other vlad at mellanox.co.il Can't compile OFED1.4.1 on SLES11 - PPC 1348 major RHEL 5 amirv at mellanox.co.il Sdp sockets doesnt closed after programs end 1472 major RHEL 5 amirv at mellanox.co.il kernel panic when trying to unload sdp or reboot 1545 major Other eli at mellanox.co.il Performance degradation in ofed 1.4.1 in TCP BW for some packets size 1287 major RHEL 5 jackm at mellanox.co.il IPoIB datagram mode initial packet loss 1506 major RHEL 5 jackm at mellanox.co.il driver occasionally fails to start on driver restart 1539 major All orenk at dev.mellanox.co.il ibutils fails to compile on ppc64 1528 major RHEL 5 jackm at mellanox.co.il IPoIB get stack when running Hadoop application. 1529 major RHEL 5 tziporet at mellanox.co.il Opensm cannot be stopped following openib failure. 1544 major SLES 10 yosefe at voltaire.com mpi_latency in ompenmpi hangs Note: There is 1.4.1 release number in bugzilla - please change bug release number to 1.4.1 if you wish it to be fixed for OFED 1.4.1 b. OFED 1.5: On Jan 26 meeting we decided that: * OFED 1.5 will be based over kernel 2.6.30 (meaning start builds when 2.6.30-rc1 is available in April) * OFED 1.5 schedule will be end of July/beginning of Aug I wish to close this again and make sure we agree to these dates. c. Open discussion Tziporet -------------- next part -------------- An HTML attachment was scrubbed... URL: From ogerlitz at Voltaire.com Thu Mar 12 08:17:50 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Thu, 12 Mar 2009 17:17:50 +0200 Subject: [ofa-general] Re: merging madeye into mainline In-Reply-To: <34D0FA0BEDF644FAA31E979C8C81C3C5@amr.corp.intel.com> References: <34D0FA0BEDF644FAA31E979C8C81C3C5@amr.corp.intel.com> Message-ID: <49B9279E.6030102@Voltaire.com> Sean Hefty wrote: > There may be a better way of doing this if we want to include > this upstream - for example, expose snooping capabilities to userspace. Yes, exposing snooping capabilities to user space and writing a user space app that does snooping sounds reasonable - what would it take to expose this capability to user-space - will it fine smoothly into the ib_umad and libibumad design/structure? Or. From dorfman.eli at gmail.com Thu Mar 12 08:37:11 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Thu, 12 Mar 2009 17:37:11 +0200 Subject: [ofa-general] Re: [PATCH 2/2] opensm: setup routing engine when in use and delete when fail In-Reply-To: <20090312132137.GB8818@sashak.voltaire.com> References: <49A6B618.1090300@gmail.com> <49A6B6EB.80700@gmail.com> <20090312132137.GB8818@sashak.voltaire.com> Message-ID: <49B92C27.7060904@gmail.com> Sasha Khapyorsky wrote: > On 17:36 Thu 26 Feb , Eli Dorfman (Voltaire) wrote: >> setup routing engine when in use and delete when fail >> setup routing engine before use. >> delete resources when routing algorithm fails >> this will save allocation for routing algorithms that are not used. >> >> Signed-off-by: Eli Dorfman >> --- >> opensm/opensm/osm_opensm.c | 20 ++++++-------------- >> opensm/opensm/osm_ucast_mgr.c | 34 +++++++++++++++++++++++++++++++++- >> 2 files changed, 39 insertions(+), 15 deletions(-) >> >> diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c >> index 7de2e5b..a2620d5 100644 >> --- a/opensm/opensm/osm_opensm.c >> +++ b/opensm/opensm/osm_opensm.c >> @@ -169,21 +169,14 @@ static void setup_routing_engine(osm_opensm_t *osm, const char *name) >> memset(re, 0, sizeof(struct osm_routing_engine)); >> >> re->name = m->name; >> - if (m->setup(re, osm)) { >> - OSM_LOG(&osm->log, OSM_LOG_VERBOSE, >> - "setup of routing" >> - " engine \'%s\' failed\n", name); >> - return; >> - } >> - OSM_LOG(&osm->log, OSM_LOG_DEBUG, >> - "\'%s\' routing engine set up\n", re->name); >> + re->setup = m->setup; >> append_routing_engine(osm, re); >> return; >> } >> } >> >> OSM_LOG(&osm->log, OSM_LOG_ERROR, >> - "cannot find or setup routing engine \'%s\'", name); >> + "cannot find or setup routing engine \'%s\'\n", name); > > This is needed fix, but unrelated to the patch. > >> } >> >> static void setup_routing_engines(osm_opensm_t *osm, const char *engine_names) >> @@ -224,18 +217,17 @@ void osm_opensm_construct(IN osm_opensm_t * const p_osm) >> >> /********************************************************************** >> **********************************************************************/ >> -static void destroy_routing_engines(osm_opensm_t *osm) >> +static void destroy_routing_engines(struct osm_routing_engine **re) > > Any reason for function prototype change? > >> { >> struct osm_routing_engine *r, *next; >> >> - next = osm->routing_engine_list; >> + next = *re; >> while (next) { >> r = next; >> next = r->next; >> - if (r->delete) >> - r->delete(r->context); > > What will happen with currently used routing engine? It will be never > deleted, leak? right? you are partially right - when opensm goes down it is not deleted. there are two RE list (prev and current). when routing engine list is updated we destroy prev used RE unless it is the same as currently used RE. see also explanation below. > >> free(r); >> } >> + *re = NULL; >> } >> >> /********************************************************************** >> @@ -289,7 +281,7 @@ void osm_opensm_destroy(IN osm_opensm_t * const p_osm) >> >> /* do the destruction in reverse order as init */ >> destroy_plugins(p_osm); >> - destroy_routing_engines(p_osm); >> + destroy_routing_engines(&p_osm->routing_engine_list); >> osm_sa_destroy(&p_osm->sa); >> osm_sm_destroy(&p_osm->sm); >> #ifdef ENABLE_OSM_PERF_MGR >> diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c >> index e404c91..7175926 100644 >> --- a/opensm/opensm/osm_ucast_mgr.c >> +++ b/opensm/opensm/osm_ucast_mgr.c >> @@ -886,7 +886,6 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) >> >> p_sw_guid_tbl = &p_mgr->p_subn->sw_guid_tbl; >> p_osm = p_mgr->p_subn->p_osm; >> - p_routing_eng = p_osm->routing_engine_list; >> >> CL_PLOCK_EXCL_ACQUIRE(p_mgr->p_lock); >> >> @@ -897,10 +896,30 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) >> ucast_mgr_setup_all_switches(p_mgr->p_subn) < 0) >> goto Exit; >> >> + /* update the entry in active list */ >> + >> p_osm->routing_engine_used = OSM_ROUTING_ENGINE_TYPE_NONE; >> + p_routing_eng = p_osm->routing_engine_list; >> while (p_routing_eng) { >> + if (!p_routing_eng->initialized && >> + p_routing_eng->setup(p_routing_eng, p_osm)) { > ^^^^^^^^ >> + OSM_LOG(p_mgr->p_log, OSM_LOG_VERBOSE, >> + "setup of routing engine \'%s\' failed\n", >> + p_routing_eng->name); > ^^^^^^^^ > > Keep indentation please. > >> + p_routing_eng = p_routing_eng->next; >> + continue; >> + } >> + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, >> + "\'%s\' routing engine set up\n", p_routing_eng->name); >> + p_routing_eng->initialized = 1; >> + >> if (!ucast_mgr_route(p_routing_eng, p_osm)) >> break; >> + >> + /* delete unused routing engine */ >> + if (p_routing_eng->delete) >> + p_routing_eng->delete(p_routing_eng->context); >> + >> p_routing_eng = p_routing_eng->next; >> } >> >> @@ -911,6 +930,19 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) >> p_osm->routing_engine_used = OSM_ROUTING_ENGINE_TYPE_MINHOP; >> } >> >> + /* if for some reason different routing engine is used */ >> + /* cleanup unused routing engine */ >> + p_routing_eng = p_osm->routing_engine_list; >> + while (p_routing_eng) { >> + if (p_routing_eng->initialized && >> + p_osm->routing_engine_used != >> + osm_routing_engine_type(p_routing_eng->name) && >> + p_routing_eng->delete) >> + p_routing_eng->delete(p_routing_eng->context); >> + >> + p_routing_eng = p_routing_eng->next; >> + } >> + > > How this section of code is useful? This code is used to free memory allocated for RE that is not used anymore. If for example we configured routing_engine=X,Z and routing_engine_used was Z Now if we modify routing_engine=Y,Z and opensm succeeds to setup Y Then we should free memory of RE(Z) that is not used now. Eli From dorfman.eli at gmail.com Thu Mar 12 08:45:19 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Thu, 12 Mar 2009 17:45:19 +0200 Subject: ***SPAM*** Re: ***SPAM*** [ofa-general] [PATCH v4] fix local port smlid In-Reply-To: <49AFE7C0.5060707@gmail.com> References: <49AFCCC0.6090507@gmail.com> <49AFE1E7.3030803@gmail.com> <49AFE65E.3080809@gmail.com> <49AFE7C0.5060707@gmail.com> Message-ID: <49B92E0F.5090507@gmail.com> Eli Dorfman (Voltaire) wrote: > fix local port smlid > > when opensm goes from master to standby due to handover its local > port smlid will be wrong (equal to its own lid). > the new master opensm will change smlid only after local sm > (previous master) is in standby and will not query its port again. > > Signed-off-by: Eli Dorfman > --- > opensm/opensm/osm_req.c | 15 +++++++++++++++ > 1 files changed, 15 insertions(+), 0 deletions(-) > > diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c > index 0865ce5..df138d7 100644 > --- a/opensm/opensm/osm_req.c > +++ b/opensm/opensm/osm_req.c > @@ -217,6 +217,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) > ib_smp_t *smp; > ib_mad_notice_attr_t *ntc; > osm_port_t *port; > + osm_port_t *smport; > ib_port_info_t *pi; > > port = osm_get_port_by_guid(sm->p_subn, sm->p_subn->sm_port_guid); > @@ -229,6 +230,20 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) > > pi = &port->p_physp->port_info; > > + smport = osm_get_port_by_guid(sm->p_subn, sm->master_sm_guid); > + if (!smport) { > + OSM_LOG(sm->p_log, OSM_LOG_ERROR, > + "ERR 1106: cannot find master SM port by guid 0x%" PRIx64 "\n", > + cl_ntoh64(sm->master_sm_guid)); > + return -1; > + } > + > + /* update local port's smlid - since we may have */ > + /* wrong value in local portinfo */ > + /* this may happen when we move to standby and master updates */ > + /* out smlid but we will not query it */ > + pi->master_sm_base_lid = smport->p_physp->port_info.base_lid; > + > /* don't bother with sending trap when SMA supports this */ > if (!local && > pi->capability_mask&(IB_PORT_CAP_HAS_TRAP|IB_PORT_CAP_HAS_CAP_NTC)) Please apply V4. The change was in testing if smport exists instead of port above. Eli From swise at opengridcomputing.com Thu Mar 12 08:41:30 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Thu, 12 Mar 2009 10:41:30 -0500 Subject: [ofa-general] Re: tcp/rdma port unification - iscsi/ddp/toe vs iwarp/ddp/toe In-Reply-To: <49B92684.4070902@Voltaire.com> References: <49B17E15.4050406@opengridcomputing.com> <49B76CD5.705@voltaire.com> <49B7D3A0.5060307@opengridcomputing.com> <15ddcffd0903110819s3b83cec0vb87ee6114496858e@mail.gmail.com> <49B7D899.1030601@opengridcomputing.com> <49B8BF05.5030004@Voltaire.com> <49B92478.4050604@opengridcomputing.com> <49B92684.4070902@Voltaire.com> Message-ID: <49B92D2A.60206@opengridcomputing.com> Or Gerlitz wrote: > Steve Wise wrote: > >> The iscsi offload drivers all maintain their own private ip addresses >> hidden from the native stack. That's how they keep the 4-tuple unique. >> > > OK, so if the ip address wasn't private then the iscsi offload arp flow > would have been the same as the iwarp one? and also the other way around, > Yes. > an iwarp stack can achieve the same impact (unique 4-tuple) either with shared > port space or with private ip address - and actually, this ip address need not > be private - it can be just --different-- from the one used for OS TCP. > Yes. If you don't hide it from the stack, then there is no way (without changing the kernel stack code) to keep the stack from using it. If you look back in the netdev/openib mailing lists, you'll see a very painful patch set I posted early on to do this sort of scheme with cxgb3. I created alias interfaces with ip addresses that were on their own subnet and the iwarp driver would only use those addresses. The downside of this patch was that its up to the sysadmin to configure everything correctly. And there was nothing to stop a TCP app from using those addresses (other than admin policy). Also it duplicates all the management of interfaces. Its a bad solution IMO. Roland also resisted admitting that solution because of these downsides. But it did resolve the issue by forcing iwarp connections on their own ip addresses. :) Here was version 3 of the patch from 2007: http://thread.gmane.org/gmane.linux.drivers.openib/44463/focus=73155 For iSCSI, its perhaps a little easier because you have only one application: iSCSI. But the original iSCSI offload submission from chelsio was to use whatever ip addresses were bound to the NIC interface and use the native neighbor discovery logic like rdmacm does. That was rejected by the netdev maintainers. > Anyway, bottom line, I see that you and Karen have chosen different approaches > for the solution of the same problem (over the same HW...) and special reason > for that - or it was a matter of how things were evolving? > > More a matter of what could advance upstream. Steve. From sashak at voltaire.com Thu Mar 12 08:49:54 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 17:49:54 +0200 Subject: [ofa-general] [PATCH v4] fix local port smlid In-Reply-To: <49AFE7C0.5060707@gmail.com> References: <49AFCCC0.6090507@gmail.com> <49AFE1E7.3030803@gmail.com> <49AFE65E.3080809@gmail.com> <49AFE7C0.5060707@gmail.com> Message-ID: <20090312154954.GU8818@sashak.voltaire.com> On 16:54 Thu 05 Mar , Eli Dorfman (Voltaire) wrote: > > fix local port smlid > > when opensm goes from master to standby due to handover its local > port smlid will be wrong (equal to its own lid). > the new master opensm will change smlid only after local sm > (previous master) is in standby and will not query its port again. I agree with an issue raising, but not fully agree with proposed fix. See below. > > Signed-off-by: Eli Dorfman > --- > opensm/opensm/osm_req.c | 15 +++++++++++++++ > 1 files changed, 15 insertions(+), 0 deletions(-) > > diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c > index 0865ce5..df138d7 100644 > --- a/opensm/opensm/osm_req.c > +++ b/opensm/opensm/osm_req.c > @@ -217,6 +217,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) > ib_smp_t *smp; > ib_mad_notice_attr_t *ntc; > osm_port_t *port; > + osm_port_t *smport; > ib_port_info_t *pi; > > port = osm_get_port_by_guid(sm->p_subn, sm->p_subn->sm_port_guid); > @@ -229,6 +230,20 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) > > pi = &port->p_physp->port_info; > > + smport = osm_get_port_by_guid(sm->p_subn, sm->master_sm_guid); > + if (!smport) { > + OSM_LOG(sm->p_log, OSM_LOG_ERROR, > + "ERR 1106: cannot find master SM port by guid 0x%" PRIx64 "\n", > + cl_ntoh64(sm->master_sm_guid)); > + return -1; > + } > + > + /* update local port's smlid - since we may have */ > + /* wrong value in local portinfo */ > + /* this may happen when we move to standby and master updates */ > + /* out smlid but we will not query it */ > + pi->master_sm_base_lid = smport->p_physp->port_info.base_lid; Normally port_info field of osm_physp structure stores PortInfo as it was last time received from the network. And we should to avoid "fixing" it this way. It could be not correct - we cannot know what master SM will really do. Instead of touching port_info I would suggest to setup destination lid using smport which you already have: madw->mad_addr.dest_lid = smport->p_physp->port_info.base_lid; Sasha From sashak at voltaire.com Thu Mar 12 08:52:41 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 17:52:41 +0200 Subject: ***SPAM*** [ofa-general] [PATCH v4] fix local port smlid In-Reply-To: <49B92E0F.5090507@gmail.com> References: <49AFCCC0.6090507@gmail.com> <49AFE1E7.3030803@gmail.com> <49AFE65E.3080809@gmail.com> <49AFE7C0.5060707@gmail.com> <49B92E0F.5090507@gmail.com> Message-ID: <20090312155241.GV8818@sashak.voltaire.com> On 17:45 Thu 12 Mar , Eli Dorfman (Voltaire) wrote: > > The change was in testing if smport exists instead of port above. I know (compared all versions already :)). My comment was "for next time". Sasha From dorfman.eli at gmail.com Thu Mar 12 08:58:17 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Thu, 12 Mar 2009 17:58:17 +0200 Subject: ***SPAM*** Re: [ofa-general] [PATCH v4] fix local port smlid In-Reply-To: <20090312154954.GU8818@sashak.voltaire.com> References: <49AFCCC0.6090507@gmail.com> <49AFE1E7.3030803@gmail.com> <49AFE65E.3080809@gmail.com> <49AFE7C0.5060707@gmail.com> <20090312154954.GU8818@sashak.voltaire.com> Message-ID: <49B93119.6000207@gmail.com> Sasha Khapyorsky wrote: > On 16:54 Thu 05 Mar , Eli Dorfman (Voltaire) wrote: >> fix local port smlid >> >> when opensm goes from master to standby due to handover its local >> port smlid will be wrong (equal to its own lid). >> the new master opensm will change smlid only after local sm >> (previous master) is in standby and will not query its port again. > > I agree with an issue raising, but not fully agree with proposed fix. > See below. > >> Signed-off-by: Eli Dorfman >> --- >> opensm/opensm/osm_req.c | 15 +++++++++++++++ >> 1 files changed, 15 insertions(+), 0 deletions(-) >> >> diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c >> index 0865ce5..df138d7 100644 >> --- a/opensm/opensm/osm_req.c >> +++ b/opensm/opensm/osm_req.c >> @@ -217,6 +217,7 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) >> ib_smp_t *smp; >> ib_mad_notice_attr_t *ntc; >> osm_port_t *port; >> + osm_port_t *smport; >> ib_port_info_t *pi; >> >> port = osm_get_port_by_guid(sm->p_subn, sm->p_subn->sm_port_guid); >> @@ -229,6 +230,20 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) >> >> pi = &port->p_physp->port_info; >> >> + smport = osm_get_port_by_guid(sm->p_subn, sm->master_sm_guid); >> + if (!smport) { >> + OSM_LOG(sm->p_log, OSM_LOG_ERROR, >> + "ERR 1106: cannot find master SM port by guid 0x%" PRIx64 "\n", >> + cl_ntoh64(sm->master_sm_guid)); >> + return -1; >> + } >> + >> + /* update local port's smlid - since we may have */ >> + /* wrong value in local portinfo */ >> + /* this may happen when we move to standby and master updates */ >> + /* out smlid but we will not query it */ >> + pi->master_sm_base_lid = smport->p_physp->port_info.base_lid; > > Normally port_info field of osm_physp structure stores PortInfo as it was > last time received from the network. And we should to avoid "fixing" it > this way. It could be not correct - we cannot know what master SM will > really do. > > Instead of touching port_info I would suggest to setup destination lid > using smport which you already have: > > madw->mad_addr.dest_lid = smport->p_physp->port_info.base_lid; > Local standby SM is polling the master SM, so if for some reason the master decides to relinquish mastership then local SM will rediscover the fabric and find the new SM. Also, it is better to update portinfo in case others will attempt to use pi->master_sm_lid. Eli From hal.rosenstock at gmail.com Thu Mar 12 08:52:08 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 12 Mar 2009 11:52:08 -0400 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] infiniband-diags/mcm_rereg_test.c: Add missing mad_rpc_close_port call In-Reply-To: <20090312145459.GR8818@sashak.voltaire.com> References: <20090311135320.GA29870@comcast.net> <20090312145459.GR8818@sashak.voltaire.com> Message-ID: On Thu, Mar 12, 2009 at 10:54 AM, Sasha Khapyorsky wrote: > On 08:53 Wed 11 Mar     , Hal Rosenstock wrote: >> >> Signed-off-by: Hal Rosenstock >> >> --- >> diff --git a/infiniband-diags/src/mcm_rereg_test.c b/infiniband-diags/src/mcm_rereg_test.c >> index de5ea90..d087e4a 100644 >> --- a/infiniband-diags/src/mcm_rereg_test.c >> +++ b/infiniband-diags/src/mcm_rereg_test.c >> @@ -492,9 +492,13 @@ int main(int argc, char **argv) >>       mad = umad_get_mad(umad); >> >>       free(umad); >> +#if 1 >> +     mad_rpc_close_port(srcport); >> +#else >>       umad_unregister(port, agent); >>       umad_close_port(port); >>       umad_done(); >> +#endif > > This is not so correct - SA class is registered directly by using > umad_register() and not with rpc. So it rather should be: > >        umad_unregister(port, agent); >        mad_rpc_close_port(srcport); > > OTOH libibmad's rpc is not used in this program (only mad formatting > stuff). So I would recommend to remove mad_rpc_* things completely. Sure; makes sense to do that if we're keeping this around. -- Hal > > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hnrose at comcast.net Thu Mar 12 09:38:56 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 12 Mar 2009 11:38:56 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] infiniband-diags/perfquery.8: Extended counters are now -x rather than -e Message-ID: <20090312163856.GA2849@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/infiniband-diags/man/perfquery.8 b/infiniband-diags/man/perfquery.8 index d8cf342..e0743d6 100644 --- a/infiniband-diags/man/perfquery.8 +++ b/infiniband-diags/man/perfquery.8 @@ -107,17 +107,17 @@ perfquery # read local port performance counters .PP perfquery 32 1 # read performance counters from lid 32, port 1 .PP -perfquery -e 32 1 # read extended performance counters from lid 32, port 1 +perfquery -x 32 1 # read extended performance counters from lid 32, port 1 .PP perfquery -a 32 # read perf counters from lid 32, all ports .PP perfquery -r 32 1 # read performance counters and reset .PP -perfquery -e -r 32 1 # read extended performance counters and reset +perfquery -x -r 32 1 # read extended performance counters and reset .PP perfquery -R 0x20 1 # reset performance counters of port 1 only .PP -perfquery -e -R 0x20 1 # reset extended performance counters of port 1 only +perfquery -x -R 0x20 1 # reset extended performance counters of port 1 only .PP perfquery -R -a 32 # reset performance counters of all ports .PP From hnrose at comcast.net Thu Mar 12 09:55:30 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 12 Mar 2009 11:55:30 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC Message-ID: <20090312165530.GA10229@comcast.net> Since these macros exit, now need to call mad_rpc_close_port prior to invoking them Also, some cosmetic formatting changes Signed-off-by: Hal Rosenstock --- diff --git a/infiniband-diags/src/grouping.c b/infiniband-diags/src/grouping.c index 0c30726..24c33ad 100644 --- a/infiniband-diags/src/grouping.c +++ b/infiniband-diags/src/grouping.c @@ -1,6 +1,7 @@ /* * Copyright (c) 2004-2007 Voltaire Inc. All rights reserved. * Copyright (c) 2007 Xsigo Systems Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -55,6 +56,7 @@ extern Node *nodesdist[MAXHOPS+1]; /* last is CA list */ extern Node *mynode; extern Port *myport; extern int maxhops_discovered; +extern struct ibmad_port *srcport; AllChassisList mylist; @@ -302,7 +304,9 @@ static void get_sfb_slot(Node *node, Port *lineport) ch->slotnum = spine4_slot_2_slb[lineport->portnum]; ch->anafanum = anafa_spine4_slot_2_slb[lineport->portnum]; } else { - IBPANIC("Unexpected node found: guid 0x%016" PRIx64, node->nodeguid); + mad_rpc_close_port(srcport); + IBPANIC("Unexpected node found: guid 0x%016" PRIx64, + node->nodeguid); } } @@ -312,8 +316,10 @@ static void get_router_slot(Node *node, Port *spineport) uint64_t guessnum = 0; if (!ch) { - if (!(node->chrecord = calloc(1, sizeof(ChassisRecord)))) + if (!(node->chrecord = calloc(1, sizeof(ChassisRecord)))) { + mad_rpc_close_port(srcport); IBPANIC("out of mem"); + } ch = node->chrecord; } @@ -345,7 +351,9 @@ static void get_router_slot(Node *node, Port *spineport) ch->slotnum = line_slot_2_sfb4[spineport->portnum]; ch->anafanum = ipr_slot_2_sfb4_port[spineport->portnum]; } else { - IBPANIC("Unexpected node found: guid 0x%016" PRIx64, spineport->node->nodeguid); + mad_rpc_close_port(srcport); + IBPANIC("Unexpected node found: guid 0x%016" PRIx64, + spineport->node->nodeguid); } } @@ -369,7 +377,9 @@ static void get_slb_slot(ChassisRecord *ch, Port *spineport) ch->slotnum = line_slot_2_sfb4[spineport->portnum]; ch->anafanum = anafa_line_slot_2_sfb4[spineport->portnum]; } else { - IBPANIC("Unexpected node found: guid 0x%016" PRIx64, spineport->node->nodeguid); + mad_rpc_close_port(srcport); + IBPANIC("Unexpected node found: guid 0x%016" PRIx64, + spineport->node->nodeguid); } } @@ -387,8 +397,10 @@ static void fill_chassis_record(Node *node) if (node->chrecord) /* somehow this node has already been passed */ return; - if (!(node->chrecord = calloc(1, sizeof(ChassisRecord)))) + if (!(node->chrecord = calloc(1, sizeof(ChassisRecord)))) { + mad_rpc_close_port(srcport); IBPANIC("out of mem"); + } ch = node->chrecord; @@ -433,8 +445,10 @@ static int get_line_index(Node *node) { int retval = 3 * (node->chrecord->slotnum - 1) + node->chrecord->anafanum; - if (retval > LINES_MAX_NUM || retval < 1) + if (retval > LINES_MAX_NUM || retval < 1) { + mad_rpc_close_port(srcport); IBPANIC("Internal error"); + } return retval; } @@ -447,8 +461,10 @@ static int get_spine_index(Node *node) else retval = node->chrecord->slotnum; - if (retval > SPINES_MAX_NUM || retval < 1) + if (retval > SPINES_MAX_NUM || retval < 1) { + mad_rpc_close_port(srcport); IBPANIC("Internal error"); + } return retval; } @@ -677,8 +693,10 @@ char *portmapstring(Port *port) static void add_chassislist() { - if (!(mylist.current = calloc(1, sizeof(ChassisList)))) + if (!(mylist.current = calloc(1, sizeof(ChassisList)))) { + mad_rpc_close_port(srcport); IBPANIC("out of mem"); + } if (mylist.first == NULL) { mylist.first = mylist.current; @@ -768,8 +786,10 @@ ChassisList *group_nodes() if (!chassis->chassisnum) chassis->chassisnum = ++chassisnum; if (!node->chrecord) { - if (!(node->chrecord = calloc(1, sizeof(ChassisRecord)))) + if (!(node->chrecord = calloc(1, sizeof(ChassisRecord)))) { + mad_rpc_close_port(srcport); IBPANIC("out of mem"); + } node->chrecord->chassisnum = chassis->chassisnum; } } diff --git a/infiniband-diags/src/ibaddr.c b/infiniband-diags/src/ibaddr.c index bb22be9..99bd33f 100644 --- a/infiniband-diags/src/ibaddr.c +++ b/infiniband-diags/src/ibaddr.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2008 Voltaire Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -145,15 +146,21 @@ int main(int argc, char **argv) if (argc) { if (ib_resolve_portid_str_via(&portid, argv[0], ibd_dest_type, - ibd_sm_id, srcport) < 0) + ibd_sm_id, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve destination port %s", argv[0]); + } } else { - if (ib_resolve_self_via(&portid, &port, 0, srcport) < 0) + if (ib_resolve_self_via(&portid, &port, 0, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve self port %s", argv[0]); + } } - if (ib_resolve_addr(&portid, port, show_lid, show_gid) < 0) + if (ib_resolve_addr(&portid, port, show_lid, show_gid) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve requested address"); + } mad_rpc_close_port(srcport); exit(0); diff --git a/infiniband-diags/src/ibnetdiscover.c b/infiniband-diags/src/ibnetdiscover.c index 25c1f7f..ecb2956 100644 --- a/infiniband-diags/src/ibnetdiscover.c +++ b/infiniband-diags/src/ibnetdiscover.c @@ -1,6 +1,7 @@ /* * Copyright (c) 2004-2008 Voltaire Inc. All rights reserved. * Copyright (c) 2007 Xsigo Systems Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -136,7 +137,6 @@ decode_port_info(void *pi, Port *port) mad_decode_field(pi, IB_PORT_LINK_SPEED_ACTIVE_F, &port->linkspeed); } - int get_port(Port *port, int portnum, ib_portid_t *portid) { @@ -146,12 +146,14 @@ get_port(Port *port, int portnum, ib_portid_t *portid) port->portnum = portnum; if (!smp_query_via(pi, portid, IB_ATTR_PORT_INFO, portnum, timeout, - srcport)) + srcport)) return -1; decode_port_info(pi, port); DEBUG("portid %s portnum %d: lid %d state %d physstate %d %s %s", - portid2str(portid), portnum, port->lid, port->state, port->physstate, get_linkwidth_str(port->linkwidth), get_linkspeed_str(port->linkspeed)); + portid2str(portid), portnum, port->lid, port->state, + port->physstate, get_linkwidth_str(port->linkwidth), + get_linkspeed_str(port->linkspeed)); return 1; } /* @@ -314,8 +316,8 @@ static void link_ports(Node *node, Port *port, Node *remotenode, Port *remoteport) { DEBUG("linking: 0x%" PRIx64 " %p->%p:%u and 0x%" PRIx64 " %p->%p:%u", - node->nodeguid, node, port, port->portnum, - remotenode->nodeguid, remotenode, remoteport, remoteport->portnum); + node->nodeguid, node, port, port->portnum, + remotenode->nodeguid, remotenode, remoteport, remoteport->portnum); if (port->remoteport) port->remoteport->remoteport = NULL; if (remoteport->remoteport) @@ -352,16 +354,20 @@ handle_port(Node *node, Port *port, ib_portid_t *path, int portnum, int dist) oldnode = find_node(&node_buf); if (oldnode) remotenode = oldnode; - else if (!(remotenode = create_node(&node_buf, path, dist + 1))) + else if (!(remotenode = create_node(&node_buf, path, dist + 1))) { + mad_rpc_close_port(srcport); IBERROR("no memory"); + } oldport = find_port(remotenode, &port_buf); if (oldport) { remoteport = oldport; if (node != remotenode || port != remoteport) IBWARN("port moving..."); - } else if (!(remoteport = create_port(remotenode, &port_buf))) + } else if (!(remoteport = create_port(remotenode, &port_buf))) { + mad_rpc_close_port(srcport); IBERROR("no memory"); + } dump_endnode(path, oldnode ? "known remote" : "new remote", remotenode, remoteport); @@ -397,14 +403,18 @@ discover(ib_portid_t *from) } node = create_node(&node_buf, from, 0); - if (!node) + if (!node) { + mad_rpc_close_port(srcport); IBERROR("out of memory"); + } mynode = node; port = create_port(node, &port_buf); - if (!port) + if (!port) { + mad_rpc_close_port(srcport); IBERROR("out of memory"); + } if (node->type != SWITCH_NODE && handle_port(node, port, from, node->localport, 0) < 0) @@ -424,7 +434,8 @@ discover(ib_portid_t *from) continue; if (get_port(&port_buf, i, path) < 0) { - IBWARN("can't reach node %s port %d", portid2str(path), i); + IBWARN("can't reach node %s port %d", + portid2str(path), i); continue; } @@ -433,8 +444,10 @@ discover(ib_portid_t *from) continue; port = create_port(node, &port_buf); - if (!port) + if (!port) { + mad_rpc_close_port(srcport); IBERROR("out of memory"); + } /* If switch, set port GUID to node GUID */ if (node->type == SWITCH_NODE) @@ -477,7 +490,7 @@ list_node(Node *node) { char *node_type; char *nodename = remap_node_name(node_name_map, node->nodeguid, - node->nodedesc); + node->nodedesc); switch(node->type) { case SWITCH_NODE: @@ -494,9 +507,8 @@ list_node(Node *node) break; } fprintf(f, "%s\t : 0x%016" PRIx64 " ports %d devid 0x%x vendid 0x%x \"%s\"\n", - node_type, - node->nodeguid, node->numports, node->devid, node->vendid, - nodename); + node_type, node->nodeguid, node->numports, node->devid, + node->vendid, nodename); free(nodename); } @@ -550,15 +562,15 @@ out_switch(Node *node, int group, char *chname) str = get_chassis_slot(node->chrecord->chassisslot); if (str) fprintf(f, "%s ", str); - fprintf(f, "%d Chip %d", node->chrecord->slotnum, node->chrecord->anafanum); + fprintf(f, "%d Chip %d", node->chrecord->slotnum, + node->chrecord->anafanum); } nodename = remap_node_name(node_name_map, node->nodeguid, - node->nodedesc); + node->nodedesc); fprintf(f, "\nSwitch\t%d %s\t\t# \"%s\" %s port 0 lid %d lmc %d\n", - node->numports, node_name(node), - nodename, + node->numports, node_name(node), nodename, node->smaenhsp0 ? "enhanced" : "base", node->smalid, node->smalmc); @@ -571,7 +583,7 @@ out_ca(Node *node, int group, char *chname) char *node_type; char *node_type2; char *nodename = remap_node_name(node_name_map, node->nodeguid, - node->nodedesc); + node->nodedesc); out_ids(node, group, chname); switch(node->type) { @@ -591,8 +603,7 @@ out_ca(Node *node, int group, char *chname) fprintf(f, "%sguid=0x%" PRIx64 "\n", node_type, node->nodeguid); fprintf(f, "%s\t%d %s\t\t# \"%s\"", - node_type2, node->numports, node_name(node), - nodename); + node_type2, node->numports, node_name(node), nodename); if (group && is_xsigo_hca(node->nodeguid)) fprintf(f, " (scp)"); fprintf(f, "\n"); @@ -627,8 +638,8 @@ out_switch_port(Port *port, int group) fprintf(f, "%s", ext_port_str); rem_nodename = remap_node_name(node_name_map, - port->remoteport->node->nodeguid, - port->remoteport->node->nodedesc); + port->remoteport->node->nodeguid, + port->remoteport->node->nodedesc); ext_port_str = out_ext_port(port->remoteport, group); fprintf(f, "\t%s[%d]%s", @@ -671,8 +682,8 @@ out_ca_port(Port *port, int group) fprintf(f, " (%" PRIx64 ") ", port->remoteport->portguid); rem_nodename = remap_node_name(node_name_map, - port->remoteport->node->nodeguid, - port->remoteport->node->nodedesc); + port->remoteport->node->nodeguid, + port->remoteport->node->nodedesc); fprintf(f, "\t\t# lid %d lmc %d \"%s\" lid %d %s%s\n", port->lid, port->lmc, rem_nodename, @@ -696,7 +707,8 @@ dump_topology(int listtype, int group) if (!listtype) { fprintf(f, "#\n# Topology file: generated on %s#\n", ctime(&t)); fprintf(f, "# Max of %d hops discovered\n", maxhops_discovered); - fprintf(f, "# Initiated from node %016" PRIx64 " port %016" PRIx64 "\n", mynode->nodeguid, mynode->portguid); + fprintf(f, "# Initiated from node %016" PRIx64 " port %016" PRIx64 "\n", + mynode->nodeguid, mynode->portguid); } /* Make pass on switches */ @@ -1007,8 +1019,10 @@ int main(int argc, char **argv) node_name_map = open_node_name_map(node_name_map_file); - if (discover(&my_portid) < 0) + if (discover(&my_portid) < 0) { + mad_rpc_close_port(srcport); IBERROR("discover"); + } if (group) chassis = group_nodes(); diff --git a/infiniband-diags/src/ibping.c b/infiniband-diags/src/ibping.c index 901079f..fb9772c 100644 --- a/infiniband-diags/src/ibping.c +++ b/infiniband-diags/src/ibping.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2008 Voltaire Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -215,22 +216,31 @@ int main(int argc, char **argv) IBERROR("Failed to open '%s' port '%d'", ibd_ca, ibd_ca_port); if (server) { - if (mad_register_server_via(ping_class, 0, 0, oui, srcport) < 0) + if (mad_register_server_via(ping_class, 0, 0, oui, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't serve class %d on this port", ping_class); + } get_host_and_domain(host_and_domain, sizeof host_and_domain); - if ((err = ibping_serv())) + if ((err = ibping_serv())) { + mad_rpc_close_port(srcport); IBERROR("ibping to %s: %s", portid2str(&portid), err); + } + mad_rpc_close_port(srcport); exit(0); } - if (mad_register_client_via(ping_class, 0, srcport) < 0) + if (mad_register_client_via(ping_class, 0, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't register ping class %d on this port", ping_class); + } if (ib_resolve_portid_str_via(&portid, argv[0], ibd_dest_type, - ibd_sm_id, srcport) < 0) + ibd_sm_id, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve destination port %s", argv[0]); + } signal(SIGINT, report); signal(SIGTERM, report); diff --git a/infiniband-diags/src/ibportstate.c b/infiniband-diags/src/ibportstate.c index 65c9ca1..c23ce84 100644 --- a/infiniband-diags/src/ibportstate.c +++ b/infiniband-diags/src/ibportstate.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2008 Voltaire Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -230,8 +231,10 @@ int main(int argc, char **argv) IBERROR("Failed to open '%s' port '%d'", ibd_ca, ibd_ca_port); if (ib_resolve_portid_str_via(&portid, argv[0], ibd_dest_type, - ibd_sm_id, srcport) < 0) + ibd_sm_id, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve destination port %s", argv[0]); + } /* First, make sure it is a switch port if it is a "set" */ if (argc >= 3) { @@ -242,24 +245,32 @@ int main(int argc, char **argv) else if (!strcmp(argv[2], "reset")) port_op = 3; else if (!strcmp(argv[2], "speed")) { - if (argc < 4) + if (argc < 4) { + mad_rpc_close_port(srcport); IBERROR("speed requires an additional parameter"); + } port_op = 4; /* Parse speed value */ speed = strtoul(argv[3], 0, 0); - if (speed > 15) + if (speed > 15) { + mad_rpc_close_port(srcport); IBERROR("invalid speed value %d", speed); + } } } err = get_node_info(&portid, data); - if (err < 0) + if (err < 0) { + mad_rpc_close_port(srcport); IBERROR("smp query nodeinfo failed"); + } if (err) { /* not switch */ if (port_op == 0) /* query op */ is_switch = 0; - else if (port_op != 4) /* other than speed op */ + else if (port_op != 4) { /* other than speed op */ + mad_rpc_close_port(srcport); IBERROR("smp query nodeinfo: Node type not switch"); + } } if (argc-1 > 0) @@ -270,8 +281,10 @@ int main(int argc, char **argv) else printf("PortInfo:\n"); err = get_port_info(&portid, data, portnum, port_op); - if (err < 0) + if (err < 0) { + mad_rpc_close_port(srcport); IBERROR("smp query portinfo failed"); + } /* Only if one of the "set" options is chosen */ if (port_op) { @@ -287,14 +300,18 @@ int main(int argc, char **argv) } err = set_port_info(&portid, data, portnum, port_op); - if (err < 0) + if (err < 0) { + mad_rpc_close_port(srcport); IBERROR("smp set portinfo failed"); + } if (port_op == 3) { /* Reset port - so also enable */ mad_set_field(data, 0, IB_PORT_PHYS_STATE_F, 2); /* Polling */ err = set_port_info(&portid, data, portnum, port_op); - if (err < 0) + if (err < 0) { + mad_rpc_close_port(srcport); IBERROR("smp set portinfo failed"); + } } } else { /* query op */ /* only compare peer port if switch port */ @@ -320,23 +337,29 @@ int main(int argc, char **argv) /* Set DrSLID to local lid */ if (ib_resolve_self_via(&selfportid, - &selfport, 0, srcport) < 0) + &selfport, 0, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("could not resolve self"); + } peerportid.drpath.drslid = (uint16_t) selfportid.lid; peerportid.drpath.drdlid = 0xffff; /* Get peer port NodeInfo to obtain peer port number */ err = get_node_info(&peerportid, data); - if (err < 0) + if (err < 0) { + mad_rpc_close_port(srcport); IBERROR("smp query nodeinfo failed"); + } mad_decode_field(data, IB_NODE_LOCAL_PORT_F, &peerlocalportnum); printf("Peer PortInfo:\n"); /* Get peer port characteristics */ err = get_port_info(&peerportid, data, peerlocalportnum, port_op); - if (err < 0) + if (err < 0) { + mad_rpc_close_port(srcport); IBERROR("smp query peer portinfofailed"); + } mad_decode_field(data, IB_PORT_LINK_WIDTH_ENABLED_F, &peerlwe ); mad_decode_field(data, IB_PORT_LINK_WIDTH_SUPPORTED_F, &peerlws); diff --git a/infiniband-diags/src/ibroute.c b/infiniband-diags/src/ibroute.c index 31b8e4c..feab823 100644 --- a/infiniband-diags/src/ibroute.c +++ b/infiniband-diags/src/ibroute.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2008 Voltaire Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -413,19 +414,25 @@ int main(int argc, char **argv) IBERROR("Failed to open '%s' port '%d'", ibd_ca, ibd_ca_port); if (!argc) { - if (ib_resolve_self_via(&portid, 0, 0, srcport) < 0) + if (ib_resolve_self_via(&portid, 0, 0, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve self addr"); + } } else if (ib_resolve_portid_str_via(&portid, argv[0], ibd_dest_type, - ibd_sm_id, srcport) < 0) + ibd_sm_id, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve destination port %s", argv[1]); + } if (multicast) err = dump_multicast_tables(&portid, startlid, endlid); else err = dump_unicast_tables(&portid, startlid, endlid); - if (err) + if (err) { + mad_rpc_close_port(srcport); IBERROR("dump tables: %s", err); + } mad_rpc_close_port(srcport); exit(0); diff --git a/infiniband-diags/src/ibsendtrap.c b/infiniband-diags/src/ibsendtrap.c index f2b3e67..72aa3f4 100644 --- a/infiniband-diags/src/ibsendtrap.c +++ b/infiniband-diags/src/ibsendtrap.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2008 Lawrence Livermore National Security + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * Produced at Lawrence Livermore National Laboratory. * Written by Ira Weiny . @@ -57,11 +58,15 @@ static int send_144_node_desc_update(void) ib_rpc_t trap_rpc; ib_mad_notice_attr_t notice; - if (ib_resolve_self_via(&selfportid, &selfport, NULL, srcport)) + if (ib_resolve_self_via(&selfportid, &selfport, NULL, srcport)) { + mad_rpc_close_port(srcport); IBERROR("can't resolve self"); + } - if (ib_resolve_smlid_via(&sm_port, 0, srcport)) + if (ib_resolve_smlid_via(&sm_port, 0, srcport)) { + mad_rpc_close_port(srcport); IBERROR("can't resolve SM destination port"); + } memset(&trap_rpc, 0, sizeof(trap_rpc)); trap_rpc.mgtclass = IB_SMI_CLASS; diff --git a/infiniband-diags/src/ibsysstat.c b/infiniband-diags/src/ibsysstat.c index 03c0515..738b1ab 100644 --- a/infiniband-diags/src/ibsysstat.c +++ b/infiniband-diags/src/ibsysstat.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2008 Voltaire Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -230,19 +231,25 @@ static char *ibsystat(ib_portid_t *portid, int attr) if (!portid->qkey) portid->qkey = IB_DEFAULT_QP1_QKEY; - if ((len = mad_build_pkt(buf, &rpc, portid, NULL, NULL)) < 0) + if ((len = mad_build_pkt(buf, &rpc, portid, NULL, NULL)) < 0) { + mad_rpc_close_port(srcport); IBPANIC("cannot build packet."); + } fd = mad_rpc_portid(srcport); agent = mad_rpc_class_agent(srcport, rpc.mgtclass); timeout = ibd_timeout ? ibd_timeout : MAD_DEF_TIMEOUT_MS; - if (umad_send(fd, agent, buf, len, timeout, 0) < 0) + if (umad_send(fd, agent, buf, len, timeout, 0) < 0) { + mad_rpc_close_port(srcport); IBPANIC("umad_send failed."); + } len = sizeof(buf) - umad_size(); - if (umad_recv(fd, buf, &len, timeout) < 0) + if (umad_recv(fd, buf, &len, timeout) < 0) { + mad_rpc_close_port(srcport); IBPANIC("umad_recv failed."); + } if (umad_status(buf)) return strerror(umad_status(buf)); @@ -342,25 +349,36 @@ int main(int argc, char **argv) IBERROR("Failed to open '%s' port '%d'", ibd_ca, ibd_ca_port); if (server) { - if (mad_register_server_via(sysstat_class, 1, 0, oui, srcport) < 0) + if (mad_register_server_via(sysstat_class, 1, 0, oui, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't serve class %d", sysstat_class); + } host_ncpu = build_cpuinfo(); - if ((err = ibsystat_serv())) + if ((err = ibsystat_serv())) { + mad_rpc_close_port(srcport); IBERROR("ibssystat to %s: %s", portid2str(&portid), err); + } + mad_rpc_close_port(srcport); exit(0); } - if (mad_register_client_via(sysstat_class, 1, srcport) < 0) + if (mad_register_client_via(sysstat_class, 1, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't register to sysstat class %d", sysstat_class); + } if (ib_resolve_portid_str_via(&portid, argv[0], ibd_dest_type, - ibd_sm_id, srcport) < 0) + ibd_sm_id, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve destination port %s", argv[0]); + } - if ((err = ibsystat(&portid, attr))) + if ((err = ibsystat(&portid, attr))) { + mad_rpc_close_port(srcport); IBERROR("ibsystat to %s: %s", portid2str(&portid), err); + } mad_rpc_close_port(srcport); exit(0); diff --git a/infiniband-diags/src/ibtracert.c b/infiniband-diags/src/ibtracert.c index a887157..ba0c1e6 100644 --- a/infiniband-diags/src/ibtracert.c +++ b/infiniband-diags/src/ibtracert.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2008 Voltaire Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -490,11 +491,15 @@ find_mcpath(ib_portid_t *from, int mlid) DEBUG("from %s", portid2str(from)); - if (!(node = calloc(1, sizeof(Node)))) + if (!(node = calloc(1, sizeof(Node)))) { + mad_rpc_close_port(srcport); IBERROR("out of memory"); + } - if (!(port = calloc(1, sizeof(Port)))) + if (!(port = calloc(1, sizeof(Port)))) { + mad_rpc_close_port(srcport); IBERROR("out of memory"); + } if (get_node(node, port, from) < 0) { IBWARN("can't reach node %s", portid2str(from)); @@ -554,8 +559,10 @@ find_mcpath(ib_portid_t *from, int mlid) if (from->drpath.cnt > 0) path->drpath.cnt--; } else { - if (!(port = calloc(1, sizeof(Port)))) + if (!(port = calloc(1, sizeof(Port)))) { + mad_rpc_close_port(srcport); IBERROR("out of memory"); + } if (get_port(port, i, path) < 0) { IBWARN("can't reach node %s port %d", portid2str(path), i); @@ -575,11 +582,15 @@ find_mcpath(ib_portid_t *from, int mlid) return 0; } - if (!(remotenode = calloc(1, sizeof(Node)))) + if (!(remotenode = calloc(1, sizeof(Node)))) { + mad_rpc_close_port(srcport); IBERROR("out of memory"); + } - if (!(remoteport = calloc(1, sizeof(Port)))) + if (!(remoteport = calloc(1, sizeof(Port)))) { + mad_rpc_close_port(srcport); IBERROR("out of memory"); + } if (get_node(remotenode, remoteport, path) < 0) { IBWARN("NodeInfo on %s port %d failed, skipping port", @@ -757,20 +768,28 @@ int main(int argc, char **argv) node_name_map = open_node_name_map(node_name_map_file); if (ib_resolve_portid_str_via(&src_portid, argv[0], ibd_dest_type, - ibd_sm_id, srcport) < 0) + ibd_sm_id, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve source port %s", argv[0]); + } if (ib_resolve_portid_str_via(&dest_portid, argv[1], ibd_dest_type, - ibd_sm_id, srcport) < 0) + ibd_sm_id, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve destination port %s", argv[1]); + } if (ibd_dest_type == IB_DEST_DRPATH) { - if (resolve_lid(&src_portid, NULL) < 0) + if (resolve_lid(&src_portid, NULL) < 0) { + mad_rpc_close_port(srcport); IBERROR("cannot resolve lid for port \'%s\'", portid2str(&src_portid)); - if (resolve_lid(&dest_portid, NULL) < 0) + } + if (resolve_lid(&dest_portid, NULL) < 0) { + mad_rpc_close_port(srcport); IBERROR("cannot resolve lid for port \'%s\'", portid2str(&dest_portid)); + } } if (dest_portid.lid == 0 || src_portid.lid == 0) { @@ -780,26 +799,35 @@ int main(int argc, char **argv) if (ibd_dest_type != IB_DEST_DRPATH) { /* first find a direct path to the src port */ - if (find_route(&my_portid, &src_portid, 0) < 0) + if (find_route(&my_portid, &src_portid, 0) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't find a route to the src port"); + } src_portid = my_portid; } if (!multicast) { - if (find_route(&src_portid, &dest_portid, dumplevel) < 0) + if (find_route(&src_portid, &dest_portid, dumplevel) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't find a route from src to dest"); + } + mad_rpc_close_port(srcport); exit(0); } else { if (mlid < 0xc000) IBWARN("invalid MLID; must be 0xc000 or larger"); } - if (!(target_portguid = find_target_portguid(&dest_portid))) + if (!(target_portguid = find_target_portguid(&dest_portid))) { + mad_rpc_close_port(srcport); IBERROR("can't reach target lid %d", dest_portid.lid); + } - if (!(endnode = find_mcpath(&src_portid, mlid))) + if (!(endnode = find_mcpath(&src_portid, mlid))) { + mad_rpc_close_port(srcport); IBERROR("can't find a multicast route from src to dest"); + } /* dump multicast path */ dump_mcpath(endnode, dumplevel); diff --git a/infiniband-diags/src/perfquery.c b/infiniband-diags/src/perfquery.c index d35e995..98b7642 100644 --- a/infiniband-diags/src/perfquery.c +++ b/infiniband-diags/src/perfquery.c @@ -1,6 +1,7 @@ /* * Copyright (c) 2004-2008 Voltaire Inc. All rights reserved. * Copyright (c) 2007 Xsigo Systems Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -262,7 +263,7 @@ static void output_aggregate_perfcounters_ext(ib_portid_t *portid) mad_dump_perfcounters_ext(buf, sizeof buf, pc, sizeof pc); - printf("# Port counters: %s port %d\n%s", portid2str(portid), ALL_PORTS, buf); + printf("# Port extended counters: %s port %d\n%s", portid2str(portid), ALL_PORTS, buf); } static void dump_perfcounters(int extended, int timeout, uint16_t cap_mask, @@ -272,10 +273,13 @@ static void dump_perfcounters(int extended, int timeout, uint16_t cap_mask, if (extended != 1) { if (!pma_query_via(pc, portid, port, timeout, - IB_GSI_PORT_COUNTERS, srcport)) + IB_GSI_PORT_COUNTERS, srcport)) { + mad_rpc_close_port(srcport); IBERROR("perfquery"); + } if (!(cap_mask & 0x1000)) { /* if PortCounters:PortXmitWait not suppported clear this counter */ + IBWARN("PortXmitWait not indicated so ignore this counter"); perf_count.xmtwait = 0; mad_encode_field(pc, IB_PC_XMT_WAIT_F, &perf_count.xmtwait); } @@ -285,31 +289,44 @@ static void dump_perfcounters(int extended, int timeout, uint16_t cap_mask, mad_dump_perfcounters(buf, sizeof buf, pc, sizeof pc); } else { if (!(cap_mask & 0x200)) /* 1.2 errata: bit 9 is extended counter support */ - IBWARN("PerfMgt ClassPortInfo 0x%x extended counters not indicated\n", cap_mask); + IBWARN("PerfMgt ClassPortInfo 0x%x extended counters not indicated", cap_mask); if (!pma_query_via(pc, portid, port, timeout, - IB_GSI_PORT_COUNTERS_EXT, srcport)) + IB_GSI_PORT_COUNTERS_EXT, srcport)) { + mad_rpc_close_port(srcport); IBERROR("perfextquery"); + } if (aggregate) aggregate_perfcounters_ext(); else mad_dump_perfcounters_ext(buf, sizeof buf, pc, sizeof pc); } - if (!aggregate) - printf("# Port counters: %s port %d\n%s", portid2str(portid), port, buf); + if (!aggregate) { + if (extended) + printf("# Port extended counters: %s port %d\n%s", + portid2str(portid), port, buf); + else + printf("# Port counters: %s port %d\n%s", + portid2str(portid), port, buf); + } } -static void reset_counters(int extended, int timeout, int mask, ib_portid_t *portid, int port) +static void reset_counters(int extended, int timeout, int mask, + ib_portid_t *portid, int port) { if (extended != 1) { if (!performance_reset_via(pc, portid, port, mask, timeout, - IB_GSI_PORT_COUNTERS, srcport)) + IB_GSI_PORT_COUNTERS, srcport)) { + mad_rpc_close_port(srcport); IBERROR("perf reset"); + } } else { if (!performance_reset_via(pc, portid, port, mask, timeout, - IB_GSI_PORT_COUNTERS_EXT, srcport)) + IB_GSI_PORT_COUNTERS_EXT, srcport)) { + mad_rpc_close_port(srcport); IBERROR("perf ext reset"); + } } } @@ -321,22 +338,28 @@ void xmt_sl_query(ib_portid_t *portid, int port, int mask) if (reset_only) { if (!performance_reset_via(pc, portid, port, mask, ibd_timeout, - IB_GSI_PORT_XMIT_DATA_SL, srcport)) + IB_GSI_PORT_XMIT_DATA_SL, srcport)) { + mad_rpc_close_port(srcport); IBERROR("perfslreset"); + } return; } if (!pma_query_via(pc, portid, port, ibd_timeout, - IB_GSI_PORT_XMIT_DATA_SL, srcport)) + IB_GSI_PORT_XMIT_DATA_SL, srcport)) { + mad_rpc_close_port(srcport); IBERROR("perfslquery"); + } mad_dump_perfcounters_xmt_sl(buf, sizeof buf, pc, sizeof pc); printf("# Port counters: %s port %d\n%s", portid2str(portid), port, buf); if(reset) if (!performance_reset_via(pc, portid, port, mask, ibd_timeout, - IB_GSI_PORT_XMIT_DATA_SL, srcport)) + IB_GSI_PORT_XMIT_DATA_SL, srcport)) { + mad_rpc_close_port(srcport); IBERROR("perfslreset"); + } } void rcv_sl_query(ib_portid_t *portid, int port, int mask) @@ -345,22 +368,28 @@ void rcv_sl_query(ib_portid_t *portid, int port, int mask) if (reset_only) { if (!performance_reset_via(pc, portid, port, mask, ibd_timeout, - IB_GSI_PORT_RCV_DATA_SL, srcport)) + IB_GSI_PORT_RCV_DATA_SL, srcport)) { + mad_rpc_close_port(srcport); IBERROR("perfslreset"); + } return; } if (!pma_query_via(pc, portid, port, ibd_timeout, - IB_GSI_PORT_RCV_DATA_SL, srcport)) + IB_GSI_PORT_RCV_DATA_SL, srcport)) { + mad_rpc_close_port(srcport); IBERROR("perfslquery"); + } mad_dump_perfcounters_rcv_sl(buf, sizeof buf, pc, sizeof pc); printf("# Port counters: %s port %d\n%s", portid2str(portid), port, buf); if(reset) if (!performance_reset_via(pc, portid, port, mask, ibd_timeout, - IB_GSI_PORT_RCV_DATA_SL, srcport)) + IB_GSI_PORT_RCV_DATA_SL, srcport)) { + mad_rpc_close_port(srcport); IBERROR("perfslreset"); + } } static int process_opt(void *context, int ch, char *optarg) @@ -396,7 +425,8 @@ static int process_opt(void *context, int ch, char *optarg) int main(int argc, char **argv) { - int mgmt_classes[4] = {IB_SMI_CLASS, IB_SMI_DIRECT_CLASS, IB_SA_CLASS, IB_PERFORMANCE_CLASS}; + int mgmt_classes[4] = {IB_SMI_CLASS, IB_SMI_DIRECT_CLASS, IB_SA_CLASS, + IB_PERFORMANCE_CLASS}; ib_portid_t portid = {0}; int mask = 0xffff; uint16_t cap_mask; @@ -450,23 +480,31 @@ int main(int argc, char **argv) if (argc) { if (ib_resolve_portid_str_via(&portid, argv[0], ibd_dest_type, - ibd_sm_id, srcport) < 0) + ibd_sm_id, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve destination port %s", argv[0]); + } } else { - if (ib_resolve_self_via(&portid, &port, 0, srcport) < 0) + if (ib_resolve_self_via(&portid, &port, 0, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve self port %s", argv[0]); + } } /* PerfMgt ClassPortInfo is a required attribute */ if (!pma_query_via(pc, &portid, port, ibd_timeout, CLASS_PORT_INFO, - srcport)) + srcport)) { + mad_rpc_close_port(srcport); IBERROR("classportinfo query"); + } /* ClassPortInfo should be supported as part of libibmad */ memcpy(&cap_mask, pc + 2, sizeof(cap_mask)); /* CapabilityMask */ cap_mask = ntohs(cap_mask); if (!(cap_mask & 0x100)) { /* bit 8 is AllPortSelect */ - if (!all_ports && port == ALL_PORTS) + if (!all_ports && port == ALL_PORTS) { + mad_rpc_close_port(srcport); IBERROR("AllPortSelect not supported"); + } if (all_ports) all_ports_loop = 1; } @@ -483,17 +521,23 @@ int main(int argc, char **argv) if (all_ports_loop || (loop_ports && (all_ports || port == ALL_PORTS))) { if (smp_query_via(data, &portid, IB_ATTR_NODE_INFO, 0, 0, - srcport) < 0) + srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("smp query nodeinfo failed"); + } node_type = mad_get_field(data, 0, IB_NODE_TYPE_F); mad_decode_field(data, IB_NODE_NPORTS_F, &num_ports); - if (!num_ports) + if (!num_ports) { + mad_rpc_close_port(srcport); IBERROR("smp query nodeinfo: num ports invalid"); + } if (node_type == IB_NODE_SWITCH) { if (smp_query_via(data, &portid, IB_ATTR_SWITCH_INFO, - 0, 0, srcport) < 0) + 0, 0, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("smp query nodeinfo failed"); + } enhancedport0 = mad_get_field(data, 0, IB_SW_ENHANCED_PORT0_F); if (enhancedport0) start_port = 0; @@ -523,15 +567,13 @@ int main(int argc, char **argv) goto done; do_reset: - if (argc <= 2 && !extended && (cap_mask & 0x1000)) mask |= (1<<16); /* reset portxmitwait */ if (all_ports_loop || (loop_ports && (all_ports || port == ALL_PORTS))) { for (i = start_port; i <= num_ports; i++) reset_counters(extended, ibd_timeout, mask, &portid, i); - } - else + } else reset_counters(extended, ibd_timeout, mask, &portid, port); done: diff --git a/infiniband-diags/src/saquery.c b/infiniband-diags/src/saquery.c index dddebc1..e5f1287 100644 --- a/infiniband-diags/src/saquery.c +++ b/infiniband-diags/src/saquery.c @@ -3,6 +3,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * Produced at Lawrence Livermore National Laboratory. * Written by Ira Weiny . @@ -99,6 +100,7 @@ struct query_cmd { static char *node_name_map_file = NULL; static nn_map_t *node_name_map = NULL; static uint64_t smkey = 1; +static struct ibmad_port *srcport; /** * Declare some globals because I don't want this to be too complex. @@ -141,8 +143,10 @@ static int sa_query(struct bind_handle *h, uint8_t method, rpc.dataoffs = IB_SA_DATA_OFFS; umad = calloc(1, len + umad_size()); - if (!umad) + if (!umad) { + mad_rpc_close_port(srcport); IBPANIC("cannot alloc mem for umad: %s\n", strerror(errno)); + } mad_build_pkt(umad, &rpc, &h->dport, NULL, data); @@ -152,9 +156,11 @@ static int sa_query(struct bind_handle *h, uint8_t method, xdump(stdout, "SA Request:\n", umad_get_mad(umad), len); ret = umad_send(h->fd, h->agent, umad, len, ibd_timeout, 0); - if (ret < 0) + if (ret < 0) { + mad_rpc_close_port(srcport); IBPANIC("umad_send failed: attr %u: %s\n", attr, strerror(errno)); + } recv_mad: ret = umad_recv(h->fd, umad, &len, ibd_timeout); @@ -163,6 +169,7 @@ recv_mad: umad = realloc(umad, umad_size() + len); goto recv_mad; } + mad_rpc_close_port(srcport); IBPANIC("umad_recv failed: attr %u: %s\n", attr, strerror(errno)); } @@ -1324,7 +1331,6 @@ static int query_mft_records(const struct query_cmd *q, bind_handle_t h, static bind_handle_t get_bind_handle(void) { - static struct ibmad_port *srcport; static struct bind_handle handle; int mgmt_classes[2] = { IB_SMI_CLASS, IB_SMI_DIRECT_CLASS }; @@ -1333,8 +1339,10 @@ static bind_handle_t get_bind_handle(void) IBERROR("Failed to open '%s' port '%d'", ibd_ca, ibd_ca_port); ib_resolve_smlid_via(&handle.dport, ibd_timeout, srcport); - if (!handle.dport.lid) + if (!handle.dport.lid) { + mad_rpc_close_port(srcport); IBPANIC("No SM found."); + } handle.dport.qp = 1; if (!handle.dport.qkey) diff --git a/infiniband-diags/src/sminfo.c b/infiniband-diags/src/sminfo.c index ebf6a47..f987445 100644 --- a/infiniband-diags/src/sminfo.c +++ b/infiniband-diags/src/sminfo.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2008 Voltaire Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -121,11 +122,15 @@ int main(int argc, char **argv) if (argc) { if (ib_resolve_portid_str_via(&portid, argv[0], ibd_dest_type, - 0, srcport) < 0) + 0, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve destination port %s", argv[0]); + } } else { - if (ib_resolve_smlid_via(&portid, ibd_timeout, srcport) < 0) + if (ib_resolve_smlid_via(&portid, ibd_timeout, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve sm port %s", argv[0]); + } } mad_encode_field(sminfo, IB_SMINFO_GUID_F, &guid); @@ -136,12 +141,15 @@ int main(int argc, char **argv) if (mod) { if (!(p = smp_set_via(sminfo, &portid, IB_ATTR_SMINFO, mod, - ibd_timeout, srcport))) - IBERROR("query"); - } else - if (!(p = smp_query_via(sminfo, &portid, IB_ATTR_SMINFO, 0, - ibd_timeout, srcport))) + ibd_timeout, srcport))) { + mad_rpc_close_port(srcport); IBERROR("query"); + } + } else if (!(p = smp_query_via(sminfo, &portid, IB_ATTR_SMINFO, 0, + ibd_timeout, srcport))) { + mad_rpc_close_port(srcport); + IBERROR("query"); + } mad_decode_field(sminfo, IB_SMINFO_GUID_F, &guid); mad_decode_field(sminfo, IB_SMINFO_ACT_F, &act); diff --git a/infiniband-diags/src/smpquery.c b/infiniband-diags/src/smpquery.c index 2ed1e65..dc6f0fa 100644 --- a/infiniband-diags/src/smpquery.c +++ b/infiniband-diags/src/smpquery.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2008 Voltaire Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -460,20 +461,28 @@ int main(int argc, char **argv) if (ibd_dest_type != IB_DEST_DRSLID) { if (ib_resolve_portid_str_via(&portid, argv[1], ibd_dest_type, - ibd_sm_id, srcport) < 0) + ibd_sm_id, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve destination port %s", argv[1]); - if ((err = fn(&portid, argv+2, argc-2))) + } + if ((err = fn(&portid, argv+2, argc-2))) { + mad_rpc_close_port(srcport); IBERROR("operation %s: %s", argv[0], err); + } } else { char concat[64]; memset(concat, 0, 64); snprintf(concat, sizeof(concat), "%s %s", argv[1], argv[2]); if (ib_resolve_portid_str_via(&portid, concat, ibd_dest_type, - ibd_sm_id, srcport) < 0) + ibd_sm_id, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve destination port %s", concat); - if ((err = fn(&portid, argv+3, argc-3))) + } + if ((err = fn(&portid, argv+3, argc-3))) { + mad_rpc_close_port(srcport); IBERROR("operation %s: %s", argv[0], err); + } } close_node_name_map(node_name_map); mad_rpc_close_port(srcport); diff --git a/infiniband-diags/src/vendstat.c b/infiniband-diags/src/vendstat.c index 0f09039..b4e3825 100644 --- a/infiniband-diags/src/vendstat.c +++ b/infiniband-diags/src/vendstat.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2008 Voltaire Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -160,17 +161,23 @@ int main(int argc, char **argv) if (argc) { if (ib_resolve_portid_str_via(&portid, argv[0], ibd_dest_type, - ibd_sm_id, srcport) < 0) + ibd_sm_id, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve destination port %s", argv[0]); + } } else { - if (ib_resolve_self_via(&portid, &port, 0, srcport) < 0) + if (ib_resolve_self_via(&portid, &port, 0, srcport) < 0) { + mad_rpc_close_port(srcport); IBERROR("can't resolve self port %s", argv[0]); + } } /* Only General Info and Port Xmit Wait Counters */ /* queries are currently supported */ - if (!general_info && !xmit_wait) + if (!general_info && !xmit_wait) { + mad_rpc_close_port(srcport); IBERROR("at least one of -N and -w must be specified"); + } /* These are Mellanox specific vendor MADs */ /* but vendors change the VendorId so how know for sure ? */ @@ -185,13 +192,18 @@ int main(int argc, char **argv) memset(&buf, 0, sizeof(buf)); /* vendor ClassPortInfo is required attribute if class supported */ call.attrid = CLASS_PORT_INFO; - if (!ib_vendor_call_via(&buf, &portid, &call, srcport)) + if (!ib_vendor_call_via(&buf, &portid, &call, srcport)) { + mad_rpc_close_port(srcport); IBERROR("classportinfo query"); + } memset(&buf, 0, sizeof(buf)); call.attrid = IB_MLX_IS3_GENERAL_INFO; - if (!ib_vendor_call_via(&buf, &portid, &call, srcport)) + if (!ib_vendor_call_via(&buf, &portid, &call, srcport)) { + mad_rpc_close_port(srcport); IBERROR("vendstat"); + } + gi = (is3_general_info_t *)&buf; if (general_info) { @@ -211,8 +223,10 @@ int main(int argc, char **argv) } if (xmit_wait) { - if (ntohs(gi->hw_info.device_id) != IS3_DEVICE_ID) + if (ntohs(gi->hw_info.device_id) != IS3_DEVICE_ID) { + mad_rpc_close_port(srcport); IBERROR("Unsupported device ID 0x%x", ntohs(gi->hw_info.device_id)); + } memset(&buf, 0, sizeof(buf)); call.attrid = IB_MLX_IS3_CONFIG_SPACE_ACCESS; @@ -222,8 +236,10 @@ int main(int argc, char **argv) cs = (is3_config_space_t *)&buf; for (i = 0; i < 16; i++) cs->record[i].address = htonl(IB_MLX_IS3_PORT_XMIT_WAIT + ((i + 1) << 12)); - if (!ib_vendor_call_via(&buf, &portid, &call, srcport)) + if (!ib_vendor_call_via(&buf, &portid, &call, srcport)) { + mad_rpc_close_port(srcport); IBERROR("vendstat"); + } for (i = 0; i < 16; i++) if (cs->record[i].data) /* PortXmitWait is 32 bit counter */ @@ -237,8 +253,10 @@ int main(int argc, char **argv) cs = (is3_config_space_t *)&buf; for (i = 0; i < 8; i++) cs->record[i].address = htonl(IB_MLX_IS3_PORT_XMIT_WAIT + ((i + 17) << 12)); - if (!ib_vendor_call_via(&buf, &portid, &call, srcport)) + if (!ib_vendor_call_via(&buf, &portid, &call, srcport)) { + mad_rpc_close_port(srcport); IBERROR("vendstat"); + } for (i = 0; i < 8; i++) if (cs->record[i].data) /* PortXmitWait is 32 bit counter */ From hal.rosenstock at gmail.com Thu Mar 12 08:59:59 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 12 Mar 2009 11:59:59 -0400 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] infiniband-diags/perfquery.8: Update man page for PortXmit/RcvDataSL In-Reply-To: <20090312143115.GM8818@sashak.voltaire.com> References: <20090310205023.GA1791@comcast.net> <20090312143115.GM8818@sashak.voltaire.com> Message-ID: On Thu, Mar 12, 2009 at 10:31 AM, Sasha Khapyorsky wrote: > On 15:50 Tue 10 Mar     , Hal Rosenstock wrote: >> >> Signed-off-by: Hal Rosenstock > > This patch does not apply against main stream, I fixed it by hand. > Please patches you post against master. I did (and do in general). Not sure why this one was broken. >Applied. Thanks. Thanks. -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From sashak at voltaire.com Thu Mar 12 09:05:28 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 18:05:28 +0200 Subject: [ofa-general] Re: [PATCH 2/2] opensm: setup routing engine when in use and delete when fail In-Reply-To: <49B92C27.7060904@gmail.com> References: <49A6B618.1090300@gmail.com> <49A6B6EB.80700@gmail.com> <20090312132137.GB8818@sashak.voltaire.com> <49B92C27.7060904@gmail.com> Message-ID: <20090312160528.GW8818@sashak.voltaire.com> On 17:37 Thu 12 Mar , Eli Dorfman (Voltaire) wrote: > > you are partially right - when opensm goes down it is not deleted. This is what I meant. > there are two RE list (prev and current). Not in this patch series. Please don't mix patches. > >> @@ -911,6 +930,19 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) > >> p_osm->routing_engine_used = OSM_ROUTING_ENGINE_TYPE_MINHOP; > >> } > >> > >> + /* if for some reason different routing engine is used */ > >> + /* cleanup unused routing engine */ > >> + p_routing_eng = p_osm->routing_engine_list; > >> + while (p_routing_eng) { > >> + if (p_routing_eng->initialized && > >> + p_osm->routing_engine_used != > >> + osm_routing_engine_type(p_routing_eng->name) && > >> + p_routing_eng->delete) > >> + p_routing_eng->delete(p_routing_eng->context); > >> + > >> + p_routing_eng = p_routing_eng->next; > >> + } > >> + > > > > How this section of code is useful? > > This code is used to free memory allocated for RE that is not used anymore. > If for example we configured routing_engine=X,Z > and routing_engine_used was Z > Now if we modify routing_engine=Y,Z But we are not modifying routing engine list now. > and opensm succeeds to setup Y > Then we should free memory of RE(Z) that is not used now. Ok. I see this may work even with static list. When one time OpenSM falls back and second time first RE succeed. Got it. Just address other comments and merge 1,2 into single patch. Sasha From sashak at voltaire.com Thu Mar 12 09:12:28 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 18:12:28 +0200 Subject: [ofa-general] [PATCH v4] fix local port smlid In-Reply-To: <49B93119.6000207@gmail.com> References: <49AFCCC0.6090507@gmail.com> <49AFE1E7.3030803@gmail.com> <49AFE65E.3080809@gmail.com> <49AFE7C0.5060707@gmail.com> <20090312154954.GU8818@sashak.voltaire.com> <49B93119.6000207@gmail.com> Message-ID: <20090312161228.GX8818@sashak.voltaire.com> On 17:58 Thu 12 Mar , Eli Dorfman (Voltaire) wrote: > > > > Instead of touching port_info I would suggest to setup destination lid > > using smport which you already have: > > > > madw->mad_addr.dest_lid = smport->p_physp->port_info.base_lid; > > > > Local standby SM is polling the master SM, so if for some reason the master > decides to relinquish mastership then local SM will rediscover the fabric and > find the new SM. Ok. So? > Also, it is better to update portinfo in case others will attempt to use pi->master_sm_lid. Using outdated PortInfo (and when SM is in Stanby it *is* outdated) is an error, so nobody should use it. Sasha From sashak at voltaire.com Thu Mar 12 09:23:11 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 18:23:11 +0200 Subject: [ofa-general] Re: [PATCH v2] opensm: set IS_SM bit during opensm init In-Reply-To: <49B918EE.5010504@gmail.com> References: <49AFC929.20606@gmail.com> <49B918EE.5010504@gmail.com> Message-ID: <20090312162311.GY8818@sashak.voltaire.com> On 16:15 Thu 12 Mar , Eli Dorfman (Voltaire) wrote: > > diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c > index 7de2e5b..cfe6474 100644 > --- a/opensm/opensm/osm_opensm.c > +++ b/opensm/opensm/osm_opensm.c > @@ -475,6 +475,11 @@ osm_opensm_bind(IN osm_opensm_t * const p_osm, IN const ib_net64_t guid) > goto Exit; > #endif /* ENABLE_OSM_PERF_MGR */ > > + /* setting IS_SM in capability mask */ > + OSM_LOG(&p_osm->log, OSM_LOG_INFO, "Setting IS_SM on port 0x%016" PRIx64 "\n", > + cl_ntoh64(guid)); > + osm_vendor_set_sm(p_osm->sm.mad_ctrl.h_bind, TRUE); This looks correct fix for me. Let's wait day or two for potential another comments. > @@ -139,16 +115,7 @@ __osm_pi_rcv_process_endport(IN osm_sm_t * sm, > } > } > > - if (port_guid == sm->p_subn->sm_port_guid) { > - /* > - We received the PortInfo for our own port. > - */ > - if (!(p_pi->capability_mask & IB_PORT_CAP_IS_SM)) > - /* > - Set the IS_SM bit to indicate our port hosts an SM. > - */ > - __osm_pi_rcv_set_sm(sm, p_physp); > - } else { > + if (port_guid != sm->p_subn->sm_port_guid) { > p_sm_tbl = &sm->p_subn->sm_guid_tbl; > if (p_pi->capability_mask & IB_PORT_CAP_IS_SM) { Just thought unrelated to this patch. In the past I started to think about redoing sm_guid_tbl so that it will keep our SM as well, for first look it should simplify some flows and SA SMinfoRecord processing. Was interrupted by something else then... Sasha From sashak at voltaire.com Thu Mar 12 09:30:13 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 18:30:13 +0200 Subject: [ofa-general] Re: [PATCH] infiniband-diags/perfquery.8: Update man page for PortXmit/RcvDataSL In-Reply-To: References: <20090310205023.GA1791@comcast.net> <20090312143115.GM8818@sashak.voltaire.com> Message-ID: <20090312163013.GZ8818@sashak.voltaire.com> On 11:59 Thu 12 Mar , Hal Rosenstock wrote: > > I did (and do in general). Not sure why this one was broken. It was was generated against some version where -e was changed to -x. Sasha From sashak at voltaire.com Thu Mar 12 09:35:34 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 18:35:34 +0200 Subject: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC In-Reply-To: <20090312165530.GA10229@comcast.net> References: <20090312165530.GA10229@comcast.net> Message-ID: <20090312163527.GA8818@sashak.voltaire.com> On 11:55 Thu 12 Mar , Hal Rosenstock wrote: > > Since these macros exit, now need to call mad_rpc_close_port > prior to invoking them Both IBERROR and IBPANIC calls exit(), it closes all opened files. This patch looks like an overkill. Sasha From sashak at voltaire.com Thu Mar 12 09:36:10 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 18:36:10 +0200 Subject: [ofa-general] Re: [PATCH] infiniband-diags/perfquery.8: Extended counters are now -x rather than -e In-Reply-To: <20090312163856.GA2849@comcast.net> References: <20090312163856.GA2849@comcast.net> Message-ID: <20090312163610.GB8818@sashak.voltaire.com> On 11:38 Thu 12 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Thu Mar 12 09:46:31 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 18:46:31 +0200 Subject: [ofa-general] Re: [PATCH] send trap144 when local priority is higher than master priority In-Reply-To: <49AFE529.7060506@gmail.com> References: <49AFE529.7060506@gmail.com> Message-ID: <20090312164631.GC8818@sashak.voltaire.com> On 16:43 Thu 05 Mar , Eli Dorfman (Voltaire) wrote: > send trap144 when local priority is higher than master priority > > send trap144 when local(standby) sm priority is higher > than master sm priority. > this will trigger heavy sweep by the master and handover of mastership. > > Signed-off-by: Eli Dorfman Applied. Thanks. Sasha From sashak at voltaire.com Thu Mar 12 10:06:51 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 19:06:51 +0200 Subject: [ofa-general] [PATCH 4/4] Many fixes on balanced minhop enhancements In-Reply-To: <49B6726E.5080706@ext.bull.net> References: <49B6726E.5080706@ext.bull.net> Message-ID: <20090312170651.GD8818@sashak.voltaire.com> Hi Nicolas, On 15:00 Tue 10 Mar , Nicolas Morey Chaisemartin wrote: > Fixed stupid order on 2nd lvl sort > Fixed bad init value for downgoing calls > Fixed bug in port counters > Fixed comments of bubble sort functions You said that those patches are RFC only and not for inclusion so this comment is "for next time" - you should fix the original patches, not to put wrong ones and then send fixes. In this way we are trying to keep our tree in always working state. Sasha From sean.hefty at intel.com Thu Mar 12 10:07:10 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 12 Mar 2009 10:07:10 -0700 Subject: ***SPAM*** Re: [ofa-general] ***SPAM*** [PATCH] infiniband-diags/mcm_rereg_test.c: Add missing mad_rpc_close_port call In-Reply-To: References: <20090311135320.GA29870@comcast.net> <20090311090337.ec293807.weiny2@llnl.gov> <0DE999D9A91E4CCBA6714985A0211242@amr.corp.intel.com> Message-ID: <17A0889B55BA49A29A7090662B57216E@amr.corp.intel.com> >Someone made the decision to want to be able to switch back and forth >earlier. This should be directed to them. It's certainly easy to >eliminate the old code. I'm wasn't suggesting that you fix the existing code, just not add to it. If someone wants to be able to switch back and forth, it makes way more sense to use an #if something_that_can_be_set_during_the_build, than #if 1, which requires source code changes in multiple places. From sean.hefty at intel.com Thu Mar 12 10:30:07 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 12 Mar 2009 10:30:07 -0700 Subject: [ofa-general] RE: merging madeye into mainline In-Reply-To: <49B9279E.6030102@Voltaire.com> References: <34D0FA0BEDF644FAA31E979C8C81C3C5@amr.corp.intel.com> <49B9279E.6030102@Voltaire.com> Message-ID: <006B7050BA0C48F6A2F0095AB922002A@amr.corp.intel.com> >Yes, exposing snooping capabilities to user space and writing a >user space app that does snooping sounds reasonable - what would >it take to expose this capability to user-space - will it fine >smoothly into the ib_umad and libibumad design/structure? libibumad needs a way for the user to indicate that they want to snoop mads, so ib_umad calls ib_register_mad_snoop(). ib_umad would also need to store copies of the mad data, rather than queuing the actual mad. I wouldn't think it would be that difficult to add, though RMPP may cause a small head-ache. (I don't remember if snooping occurs before or after RMPP packets are reassembled. If it's before, it'll be easier to copy the mad.) - Sean From hal.rosenstock at gmail.com Thu Mar 12 10:31:23 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 12 Mar 2009 13:31:23 -0400 Subject: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC In-Reply-To: <20090312163527.GA8818@sashak.voltaire.com> References: <20090312165530.GA10229@comcast.net> <20090312163527.GA8818@sashak.voltaire.com> Message-ID: On Thu, Mar 12, 2009 at 12:35 PM, Sasha Khapyorsky wrote: > On 11:55 Thu 12 Mar     , Hal Rosenstock wrote: >> >> Since these macros exit, now need to call mad_rpc_close_port >> prior to invoking them > > Both IBERROR and IBPANIC calls exit(), it closes all opened files. This > patch looks like an overkill. It's not a matter of relying on exit for open fds but rather the allocated memory under the covers of mad_rpc_open_port so no longer can one rely on just exit and this needs to be made explicit. -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From sean.hefty at intel.com Thu Mar 12 10:35:58 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 12 Mar 2009 10:35:58 -0700 Subject: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC In-Reply-To: References: <20090312165530.GA10229@comcast.net> <20090312163527.GA8818@sashak.voltaire.com> Message-ID: >It's not a matter of relying on exit for open fds but rather the >allocated memory under the covers of mad_rpc_open_port so no longer >can one rely on just exit and this needs to be made explicit. The OS should reclaim any allocated memory not freed by the app when it exits. Is this your concern? From hal.rosenstock at gmail.com Thu Mar 12 10:45:22 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 12 Mar 2009 13:45:22 -0400 Subject: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC In-Reply-To: References: <20090312165530.GA10229@comcast.net> <20090312163527.GA8818@sashak.voltaire.com> Message-ID: On Thu, Mar 12, 2009 at 1:35 PM, Sean Hefty wrote: >>It's not a matter of relying on exit for open fds but rather the >>allocated memory under the covers of mad_rpc_open_port so no longer >>can one rely on just exit and this needs to be made explicit. > > The OS should reclaim any allocated memory not freed by the app when it exits. > Is this your concern? malloc'd memory ? From sashak at voltaire.com Thu Mar 12 10:53:03 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 19:53:03 +0200 Subject: [ofa-general] Re: [PATCH] opensm/osm_ucast_ftree.c: remove __osm_ftree prefix in static functions In-Reply-To: <49B8D62A.6060107@dev.mellanox.co.il> References: <49B8D62A.6060107@dev.mellanox.co.il> Message-ID: <20090312175303.GF8818@sashak.voltaire.com> On 11:30 Thu 12 Mar , Yevgeny Kliteynik wrote: > Hi Sasha, > > Removing all the __osm_ftree prefixes in static functions naming. > > Signed-off-by: Yevgeny Kliteynik Applied. Thanks. I also formatted osm_ucast_ftree.c with osm_indent. Sasha From sashak at voltaire.com Thu Mar 12 11:07:48 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 20:07:48 +0200 Subject: [ofa-general] Re: [PATCH v2] opensm/osm_link_mgr.c initialize SMSL In-Reply-To: <49B634E3.7080006@Sun.COM> References: <49B634E3.7080006@Sun.COM> Message-ID: <20090312180741.GG8818@sashak.voltaire.com> On 10:37 Tue 10 Mar , Line.Holen at Sun.COM wrote: > The SMSL is currently not initialized. Whatever is written in PortInfo > at power up is kept. > > The lash routing algorithm makes active use of SL to avoid deadlock in > the fabric. The endnodes needs to be instructed to use the SL calculated > for the endnode-to-SMnode path for SA traffic. If not, the SA traffic > can cause deadlock. > > This change makes sure that the SMSL is properly initialized on all > ports. If flash routing is used, the SMSL is set up based on routing > calculations. For other routing algorithms the default SMSL is used. > > > Signed-off-by: Line Holen Applied. Thanks. Sasha From hal.rosenstock at gmail.com Thu Mar 12 11:05:09 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 12 Mar 2009 14:05:09 -0400 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC In-Reply-To: References: <20090312165530.GA10229@comcast.net> <20090312163527.GA8818@sashak.voltaire.com> Message-ID: On Thu, Mar 12, 2009 at 1:45 PM, Hal Rosenstock wrote: > On Thu, Mar 12, 2009 at 1:35 PM, Sean Hefty wrote: >>>It's not a matter of relying on exit for open fds but rather the >>>allocated memory under the covers of mad_rpc_open_port so no longer >>>can one rely on just exit and this needs to be made explicit. >> >> The OS should reclaim any allocated memory not freed by the app when it exits. >> Is this your concern? > > malloc'd memory ? Many OSs do but not all. Should we rely on this ? Also, don't other userspace programs attempt to properly cleanup after themselves rather than rely on OS cleanup behavior ? From sashak at voltaire.com Thu Mar 12 11:19:51 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Thu, 12 Mar 2009 20:19:51 +0200 Subject: [ofa-general] Re: [PATCH 1/2] opensm: Add common ib_gid_is_notzero routine In-Reply-To: <20090310192234.GA6431@comcast.net> References: <20090310192234.GA6431@comcast.net> Message-ID: <20090312181951.GH8818@sashak.voltaire.com> On 14:22 Tue 10 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock > --- > diff --git a/opensm/include/iba/ib_types.h b/opensm/include/iba/ib_types.h > index e7cbb79..bfe8665 100644 > --- a/opensm/include/iba/ib_types.h > +++ b/opensm/include/iba/ib_types.h > @@ -2338,6 +2338,40 @@ ib_gid_get_guid(IN const ib_gid_t * const p_gid) > * ib_gid_t > *********/ > > +OSM_EXPORT const ib_gid_t ib_zero_gid; > + > +/****f* IBA Base: Types/ib_gid_is_notzero > +* NAME > +* ib_gid_is_notzero > +* > +* DESCRIPTION > +* Returns a boolean indicating whether or not the GID is zero. > +* > +* SYNOPSIS > +*/ > +static inline boolean_t OSM_API > +ib_gid_is_notzero(IN const ib_gid_t * const p_gid) > +{ > + if (memcmp(p_gid, &ib_zero_gid, sizeof(*p_gid))) > + return TRUE; > + return FALSE; > +} [snip...] > diff --git a/opensm/opensm/osm_helper.c b/opensm/opensm/osm_helper.c > index c56f5b2..b40ba0c 100644 > --- a/opensm/opensm/osm_helper.c > +++ b/opensm/opensm/osm_helper.c > @@ -535,6 +535,8 @@ const char *ib_get_trap_str(ib_net16_t trap_num) > return "Unknown"; > } > > +const ib_gid_t ib_zero_gid = {}; > + This introduces dependency against libopensm for ib_types.h users. I would suggest to put all this stuff to osm_helper. Sasha From weiny2 at llnl.gov Thu Mar 12 11:23:07 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Thu, 12 Mar 2009 11:23:07 -0700 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC In-Reply-To: References: <20090312165530.GA10229@comcast.net> <20090312163527.GA8818@sashak.voltaire.com> Message-ID: <20090312112307.972a44ec.weiny2@llnl.gov> On Thu, 12 Mar 2009 14:05:09 -0400 Hal Rosenstock wrote: > On Thu, Mar 12, 2009 at 1:45 PM, Hal Rosenstock > wrote: > > On Thu, Mar 12, 2009 at 1:35 PM, Sean Hefty wrote: > >>>It's not a matter of relying on exit for open fds but rather the > >>>allocated memory under the covers of mad_rpc_open_port so no longer > >>>can one rely on just exit and this needs to be made explicit. > >> > >> The OS should reclaim any allocated memory not freed by the app when it exits. > >> Is this your concern? > > > > malloc'd memory ? > > Many OSs do but not all. Should we rely on this ? Also, don't other > userspace programs attempt to properly cleanup after themselves rather > than rely on OS cleanup behavior ? Strictly speaking I think Hal is right on this one. I should have made the change myself was lazy. Another more tangible reason for this is we may change the behavior of the macros in the future. IBERROR recently fooled Sasha into thinking execution continued. IBPANIC is perhaps a bit more clear that it will exit. Ira > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http:// lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http:// openib.org/mailman/listinfo/openib-general > -- Ira Weiny Math Programer/Computer Scientist Larence Livermore National Lab weiny2 at llnl.gov From eitan at mellanox.co.il Thu Mar 12 11:06:17 2009 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Thu, 12 Mar 2009 20:06:17 +0200 Subject: [ofa-general] RE: ibdmchk: How to generate PSL and SL2VL files ? In-Reply-To: <829ded920903112326s34486709s67538f900d6e5943@mail.gmail.com> Message-ID: <40FA0A8088E8A441973D37502F00933E0140FCFE@mtlexch01.mtl.com> Hi Mahesh, The help message is erroneous: First these file are not mandatory. Second OpenSM does not generate them. The files can be generated by performing SA PathRecord queries. We do not automation for that yet. So it seems we need to perform 2 things: 1. Fix the help / man 2. Write some SA client code for obtaining the data. Thanks for pointing this up. Eitan Zahavi Senior Engineering Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL > -----Original Message----- > From: Keshetti Mahesh [mailto:keshetti.mahesh at gmail.com] > Sent: Thursday, March 12, 2009 8:27 AM > To: Eitan Zahavi > Cc: openIB > Subject: Re: ibdmchk: How to generate PSL and SL2VL files ? > > On Thu, Mar 12, 2009 at 11:52 AM, Keshetti Mahesh > wrote: > > Hi, > > > > Current implementation of 'ibdmchk' in cluster verification mode > > requires path<->SL(PSL) and SL2VL files as input. And the help(-h) > > message says that these files can be generated by OpenSM > with '-D 0x43'. > > > > But I am not able to generate any such file with OpenSM. And there > > isn't any information related to these files in man page as well as > > help(-h). Can anyone please tell me how to generate > path<->SL and SL2VL files using OpenSM? > > > > I am using OpenSM of OFED-1.4 distribution. > > -Mahesh > From hnrose at comcast.net Thu Mar 12 13:19:18 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 12 Mar 2009 15:19:18 -0500 Subject: [ofa-general] ***SPAM*** [PATCHv2 1/2] opensm: Add common ib_gid_is_notzero routine Message-ID: <20090312201918.GA25024@comcast.net> Signed-off-by: Hal Rosenstock --- Changes since v1: Moved ib_gid_is_notzero from ib_types.h to osm_helper.h diff --git a/opensm/include/opensm/osm_helper.h b/opensm/include/opensm/osm_helper.h index 9222853..cbb6408 100644 --- a/opensm/include/opensm/osm_helper.h +++ b/opensm/include/opensm/osm_helper.h @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -168,6 +169,40 @@ const char *ib_get_trap_str(uint16_t trap_num); * *********/ +OSM_EXPORT const ib_gid_t ib_zero_gid; + +/****f* IBA Base: Types/ib_gid_is_notzero +* NAME +* ib_gid_is_notzero +* +* DESCRIPTION +* Returns a boolean indicating whether or not the GID is zero. +* +* SYNOPSIS +*/ +static inline boolean_t OSM_API +ib_gid_is_notzero(IN const ib_gid_t * const p_gid) +{ + if (memcmp(p_gid, &ib_zero_gid, sizeof(*p_gid))) + return TRUE; + return FALSE; +} + +/* +* PARAMETERS +* p_gid +* [in] Pointer to the GID object. +* +* RETURN VALUES +* Returns TRUE if GID is not zero. +* FALSE otherwise. +* +* NOTES +* +* SEE ALSO +* ib_gid_t +*********/ + /****f* OpenSM: Helper/osm_dump_port_info * NAME * osm_dump_port_info diff --git a/opensm/opensm/libopensm.map b/opensm/opensm/libopensm.map index 7cd2aba..9cfa886 100644 --- a/opensm/opensm/libopensm.map +++ b/opensm/opensm/libopensm.map @@ -18,6 +18,7 @@ OPENSM_1.5 { ib_get_sm_attr_str; ib_get_sa_attr_str; ib_get_trap_str; + ib_zero_gid; osm_dump_port_info; osm_dump_portinfo_record; osm_dump_guidinfo_record; diff --git a/opensm/opensm/osm_helper.c b/opensm/opensm/osm_helper.c index c56f5b2..b40ba0c 100644 --- a/opensm/opensm/osm_helper.c +++ b/opensm/opensm/osm_helper.c @@ -535,6 +535,8 @@ const char *ib_get_trap_str(ib_net16_t trap_num) return "Unknown"; } +const ib_gid_t ib_zero_gid = {}; + /********************************************************************** **********************************************************************/ static ib_api_status_t From hnrose at comcast.net Thu Mar 12 13:21:34 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 12 Mar 2009 15:21:34 -0500 Subject: [ofa-general] ***SPAM*** [PATCHv2] opensm/PerfMgr: Better redirection support Message-ID: <20090312202134.GC25024@comcast.net> Handle PKey and QPN redirection information GID redirection handling remains Signed-off-by: Hal Rosenstock --- Changes since v1: Added include of osm_helper.h to osm_perfmgr.c Notes: osm_console redirection patch is unchanged Also, relies on ib_gid_is_notzero patch diff --git a/opensm/include/opensm/osm_perfmgr.h b/opensm/include/opensm/osm_perfmgr.h index 45dec54..651ea80 100644 --- a/opensm/include/opensm/osm_perfmgr.h +++ b/opensm/include/opensm/osm_perfmgr.h @@ -92,8 +92,14 @@ typedef enum { /* Redirection information */ typedef struct redir { + boolean_t redirection; + boolean_t invalid; + ib_gid_t redir_gid; ib_net16_t redir_lid; + ib_net16_t redir_pkey; ib_net32_t redir_qp; + uint16_t redir_pkey_ix; + ib_net16_t orig_lid; } redir_t; /* Node to store information about which nodes we are monitoring */ @@ -134,6 +140,7 @@ typedef struct osm_perfmgr { uint32_t max_outstanding_queries; cl_qmap_t monitored_map; /* map the nodes we are tracking */ __monitored_node_t *remove_list; + int16_t local_port; } osm_perfmgr_t; /* * FIELDS diff --git a/opensm/opensm/osm_perfmgr.c b/opensm/opensm/osm_perfmgr.c index 4a6f65c..fc1b7cb 100644 --- a/opensm/opensm/osm_perfmgr.c +++ b/opensm/opensm/osm_perfmgr.c @@ -47,7 +47,6 @@ #endif /* HAVE_CONFIG_H */ #ifdef ENABLE_OSM_PERF_MGR - #include #include #include @@ -65,8 +64,11 @@ #include #include #include +#include #define OSM_PERFMGR_INITIAL_TID_VALUE 0xcafe +#define MAX_LOCAL_IBPORTS 64 +#define MAX_LOCAL_PKEYS 256 #if ENABLE_OSM_PERF_MGR_PROFILE struct { @@ -118,8 +120,6 @@ static inline void diff_time(struct timeval *before, } #endif -extern int wait_for_pending_transactions(osm_stats_t * stats); - /********************************************************************** * Internal helper functions. **********************************************************************/ @@ -203,6 +203,7 @@ osm_perfmgr_mad_send_err_callback(void *bind_context, osm_madw_t * p_madw) uint8_t port = context->perfmgr_context.port; cl_map_item_t *p_node; __monitored_node_t *p_mon_node; + ib_net16_t orig_lid; OSM_LOG_ENTER(pm->log); @@ -233,9 +234,10 @@ osm_perfmgr_mad_send_err_callback(void *bind_context, osm_madw_t * p_madw) p_mon_node->guid, p_mon_node->redir_tbl_size); goto Exit; } - /* Clear redirection info */ - p_mon_node->redir_port[port].redir_lid = 0; - p_mon_node->redir_port[port].redir_qp = 0; + /* Clear redirection info for this port except orig_lid */ + orig_lid = p_mon_node->redir_port[port].orig_lid; + memset(&p_mon_node->redir_port[port], 0, sizeof(redir_t)); + p_mon_node->redir_port[port].orig_lid = orig_lid; cl_plock_release(pm->lock); } @@ -292,6 +294,40 @@ osm_perfmgr_bind(osm_perfmgr_t * const pm, const ib_net64_t port_guid) goto Exit; } + /* if redirection enabled, determine local port from port GUID */ + if (pm->subn->opt.perfmgr_redir) { + ib_port_attr_t attr_array[MAX_LOCAL_IBPORTS]; + uint32_t num_ports = MAX_LOCAL_IBPORTS; + int i; + + for (i = 0; i < num_ports; i++) { + attr_array[i].num_pkeys = 0; + attr_array[i].p_pkey_table = NULL; + } + + /* call transport layer for a list of local port GUIDs */ + status = osm_vendor_get_all_port_attr(pm->subn->p_osm->p_vendor, + attr_array, &num_ports); + if (status != IB_SUCCESS) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C1C: osm_vendor_get_all_port_attr status 0x%x\n", + status); + goto Exit; + } + if (num_ports == 0) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C1D: No local ports detected!\n"); + goto Exit; + } + + for (i = 0; i < num_ports; i++) { + if (port_guid == attr_array[i].port_guid) { + pm->local_port = attr_array[i].port_num; + break; + } + } + } + Exit: OSM_LOG_EXIT(pm->log); return (status); @@ -321,25 +357,16 @@ static ib_net32_t get_qp(__monitored_node_t * mon_node, uint8_t port) if (mon_node && mon_node->redir_tbl_size && port < mon_node->redir_tbl_size && - mon_node->redir_port[port].redir_lid && + mon_node->redir_port[port].redirection && mon_node->redir_port[port].redir_qp) qp = mon_node->redir_port[port].redir_qp; return qp; } -/********************************************************************** - * Given a node, a port, and an optional monitored node, - * return the appropriate lid to query that port - **********************************************************************/ static ib_net16_t -get_lid(osm_node_t * p_node, uint8_t port, __monitored_node_t * mon_node) +get_base_lid(osm_node_t * p_node, uint8_t port) { - if (mon_node && mon_node->redir_tbl_size && - port < mon_node->redir_tbl_size && - mon_node->redir_port[port].redir_lid) - return mon_node->redir_port[port].redir_lid; - switch (p_node->node_info.node_type) { case IB_NODE_TYPE_CA: case IB_NODE_TYPE_ROUTER: @@ -352,12 +379,27 @@ get_lid(osm_node_t * p_node, uint8_t port, __monitored_node_t * mon_node) } /********************************************************************** + * Given a node, a port, and an optional monitored node, + * return the lid appropriate to query that port + **********************************************************************/ +static ib_net16_t +get_lid(osm_node_t * p_node, uint8_t port, __monitored_node_t * mon_node) +{ + if (mon_node && mon_node->redir_tbl_size && + port < mon_node->redir_tbl_size && + mon_node->redir_port[port].redir_lid) + return mon_node->redir_port[port].redir_lid; + + return get_base_lid(p_node, port); +} + +/********************************************************************** * Form and send the Port Counters MAD for a single port. **********************************************************************/ static ib_api_status_t osm_perfmgr_send_pc_mad(osm_perfmgr_t * perfmgr, ib_net16_t dest_lid, - ib_net32_t dest_qp, uint8_t port, uint8_t mad_method, - osm_madw_context_t * const p_context) + ib_net32_t dest_qp, uint16_t pkey_ix, uint8_t port, + uint8_t mad_method, osm_madw_context_t * const p_context) { ib_api_status_t status = IB_SUCCESS; ib_port_counters_t *port_counter = NULL; @@ -396,8 +438,7 @@ osm_perfmgr_send_pc_mad(osm_perfmgr_t * perfmgr, ib_net16_t dest_lid, p_madw->mad_addr.addr_type.gsi.remote_qp = dest_qp; p_madw->mad_addr.addr_type.gsi.remote_qkey = cl_hton32(IB_QP1_WELL_KNOWN_Q_KEY); - /* FIXME what about other partitions */ - p_madw->mad_addr.addr_type.gsi.pkey_ix = 0; + p_madw->mad_addr.addr_type.gsi.pkey_ix = pkey_ix; p_madw->mad_addr.addr_type.gsi.service_level = 0; p_madw->mad_addr.addr_type.gsi.global_route = FALSE; p_madw->resp_expected = TRUE; @@ -432,28 +473,32 @@ static void __collect_guids(cl_map_item_t * const p_map_item, void *context) uint64_t node_guid = cl_ntoh64(node->node_info.node_guid); osm_perfmgr_t *pm = (osm_perfmgr_t *) context; __monitored_node_t *mon_node = NULL; - uint32_t size; + uint32_t num_ports; + int port; OSM_LOG_ENTER(pm->log); if (cl_qmap_get(&pm->monitored_map, node_guid) == cl_qmap_end(&pm->monitored_map)) { /* if not already in our map add it */ - size = osm_node_get_num_physp(node); - mon_node = malloc(sizeof(*mon_node) + sizeof(redir_t) * size); + num_ports = osm_node_get_num_physp(node); + mon_node = malloc(sizeof(*mon_node) + sizeof(redir_t) * num_ports); if (!mon_node) { OSM_LOG(pm->log, OSM_LOG_ERROR, "PerfMgr: ERR 4C06: " "malloc failed: not handling node %s" "(GUID 0x%" PRIx64 ")\n", node->print_desc, node_guid); goto Exit; } - memset(mon_node, 0, sizeof(*mon_node) + sizeof(redir_t) * size); + memset(mon_node, 0, sizeof(*mon_node) + sizeof(redir_t) * num_ports); mon_node->guid = node_guid; mon_node->name = strdup(node->print_desc); - mon_node->redir_tbl_size = size; + mon_node->redir_tbl_size = num_ports; /* check for enhanced switch port 0 */ mon_node->esp0 = (node->sw && ib_switch_info_is_enhanced_port0(&node->sw->switch_info)); + for (port = (mon_node->esp0) ? 0 : 1; port < num_ports; port++) + mon_node->redir_port[port].orig_lid = get_base_lid(node, port); + cl_qmap_insert(&(pm->monitored_map), node_guid, (cl_map_item_t *) mon_node); } @@ -511,6 +556,10 @@ __osm_perfmgr_query_counters(cl_map_item_t * const p_map_item, void *context) if (!osm_node_get_physp_ptr(node, port)) continue; + if (mon_node->redir_port[port].redirection && + mon_node->redir_port[port].invalid) + continue; + lid = get_lid(node, port, mon_node); if (lid == 0) { OSM_LOG(pm->log, OSM_LOG_DEBUG, "WARN: node 0x%" PRIx64 @@ -532,8 +581,10 @@ __osm_perfmgr_query_counters(cl_map_item_t * const p_map_item, void *context) PRIx64 " port %d (lid %u) (%s)\n", node_guid, port, cl_ntoh16(lid), node->print_desc); status = - osm_perfmgr_send_pc_mad(pm, lid, remote_qp, port, - IB_MAD_METHOD_GET, &mad_context); + osm_perfmgr_send_pc_mad(pm, lid, remote_qp, + mon_node->redir_port[port].redir_pkey_ix, + port, IB_MAD_METHOD_GET, + &mad_context); if (status != IB_SUCCESS) OSM_LOG(pm->log, OSM_LOG_ERROR, "ERR 4C09: " "Failed to issue port counter query for node 0x%" @@ -550,6 +601,7 @@ Exit: * Discovery stuff. * Basically this code should not be here, but merged with main OpenSM **********************************************************************/ +extern int wait_for_pending_transactions(osm_stats_t * stats); extern void osm_drop_mgr_process(IN osm_sm_t *sm); static int sweep_hop_1(osm_sm_t * sm) @@ -980,6 +1032,10 @@ osm_perfmgr_check_overflow(osm_perfmgr_t * pm, __monitored_node_t *mon_node, osm_node_t *p_node = NULL; ib_net16_t lid = 0; + if (mon_node->redir_port[port].redirection && + mon_node->redir_port[port].invalid) + goto Exit; + osm_log(pm->log, OSM_LOG_VERBOSE, "PerfMgr: Counter overflow: %s (0x%" PRIx64 ") port %d; clearing counters\n", @@ -1004,8 +1060,10 @@ osm_perfmgr_check_overflow(osm_perfmgr_t * pm, __monitored_node_t *mon_node, mad_context.perfmgr_context.mad_method = IB_MAD_METHOD_SET; /* clear port counters */ status = - osm_perfmgr_send_pc_mad(pm, lid, remote_qp, port, - IB_MAD_METHOD_SET, &mad_context); + osm_perfmgr_send_pc_mad(pm, lid, remote_qp, + mon_node->redir_port[port].redir_pkey_ix, + port, IB_MAD_METHOD_SET, + &mad_context); if (status != IB_SUCCESS) OSM_LOG(pm->log, OSM_LOG_ERROR, "PerfMgr: ERR 4C11: " "Failed to send clear counters MAD for %s (0x%" @@ -1063,6 +1121,73 @@ osm_perfmgr_log_events(osm_perfmgr_t * pm, __monitored_node_t *mon_node, uint8_t time_diff, mon_node->name, mon_node->guid, port); } +static boolean_t validate_redir_pkey(osm_perfmgr_t *pm, ib_net16_t pkey, + uint16_t *pkey_ix) +{ + ib_port_attr_t attr_array[MAX_LOCAL_IBPORTS]; + uint32_t num_ports = MAX_LOCAL_IBPORTS; + ib_api_status_t status; + boolean_t sts = FALSE; + uint16_t i = 0; + + OSM_LOG_ENTER(pm->log); + + for (i = 0; i < num_ports; i++) { + attr_array[i].num_pkeys = 0; + attr_array[i].p_pkey_table = NULL; + } + + /* If local port couldn't be determined previously */ + if (pm->local_port == -1) + goto not_found; + + attr_array[pm->local_port].num_pkeys = MAX_LOCAL_PKEYS; + attr_array[pm->local_port].p_pkey_table = + malloc(MAX_LOCAL_PKEYS * sizeof(ib_net16_t)); + if (!attr_array[pm->local_port].p_pkey_table) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C20: No memory for port %d pkey table\n", + pm->local_port); + goto not_found; + } + + /* call the transport layer for a list of local port pkeys */ + status = osm_vendor_get_all_port_attr(pm->subn->p_osm->p_vendor, + attr_array, &num_ports); + if (status != IB_SUCCESS) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C1E: osm_vendor_get_all_port_attr status 0x%x\n", + status); + goto not_found; + } + if (num_ports == 0 || pm->local_port > num_ports) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C1F: No local ports detected or local port out of range!\n"); + goto not_found; + } + ib_net16_t *pkey_table = attr_array[pm->local_port].p_pkey_table; + for (i = 0; i < attr_array[pm->local_port].num_pkeys; i++) + if (pkey_table[i] == pkey) + break; + if (i == attr_array[pm->local_port].num_pkeys) { + i = 0; + goto not_found; + } + free(attr_array[pm->local_port].p_pkey_table); + sts = TRUE; + goto Exit; + +not_found: + if (attr_array[pm->local_port].p_pkey_table) + free(attr_array[pm->local_port].p_pkey_table); + sts = FALSE; +Exit: + if (pkey_ix) + *pkey_ix = i; + OSM_LOG_EXIT(pm->log); + return sts; +} + /********************************************************************** * The dispatcher uses a thread pool which will call this function when * we have a thread available to process our mad received from the wire. @@ -1082,6 +1207,8 @@ static void osm_pc_rcv_process(void *context, void *data) perfmgr_db_data_cnt_reading_t data_reading; cl_map_item_t *p_node; __monitored_node_t *p_mon_node; + uint16_t pkey_ix; + boolean_t invalid = FALSE; OSM_LOG_ENTER(pm->log); @@ -1105,7 +1232,8 @@ static void osm_pc_rcv_process(void *context, void *data) p_mad->attr_id == IB_MAD_ATTR_CLASS_PORT_INFO); /* Response could also be redirection (IBM eHCA PMA does this) */ - if (p_mad->attr_id == IB_MAD_ATTR_CLASS_PORT_INFO) { + if (p_mad->status & IB_MAD_STATUS_REDIRECT && + p_mad->attr_id == IB_MAD_ATTR_CLASS_PORT_INFO) { char gid_str[INET6_ADDRSTRLEN]; ib_class_port_info_t *cpi = (ib_class_port_info_t *) & @@ -1119,17 +1247,48 @@ static void osm_pc_rcv_process(void *context, void *data) sizeof gid_str), cl_ntoh32(cpi->redir_qp)); - /* LID or GID redirection ? */ - /* For GID redirection, need to get PathRecord from SA */ + /* valid redirection ? */ if (cpi->redir_lid == 0) { - OSM_LOG(pm->log, OSM_LOG_VERBOSE, - "GID redirection not currently implemented!\n"); - goto Exit; + if (!ib_gid_is_notzero(&cpi->redir_gid)) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C17: Invalid redirection " + "(both redirect LID and GID are zero)\n"); + invalid = TRUE; + } + } + if (cpi->redir_qp == 0) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C18: Invalid RedirectQP\n"); + invalid = TRUE; + } + if (cpi->redir_pkey == 0) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C19: Invalid RedirectP_Key\n"); + invalid = TRUE; + } + if (cpi->redir_qkey != IB_QP1_WELL_KNOWN_Q_KEY) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C1A: Invalid RedirectQ_Key\n"); + invalid = TRUE; + } + + if (!validate_redir_pkey(pm, cpi->redir_pkey, &pkey_ix)) { + OSM_LOG(pm->log, OSM_LOG_ERROR, + "ERR 4C1B: Index for Pkey 0x%x not found\n", + cl_ntoh16(cpi->redir_pkey)); + invalid = TRUE; } if (!pm->subn->opt.perfmgr_redir) { - OSM_LOG(pm->log, OSM_LOG_ERROR, "ERR 4C16: " - "redirection requested but disabled\n"); + OSM_LOG(pm->log, OSM_LOG_ERROR, "ERR 4C16: " + "redirection requested but disabled\n"); + invalid = TRUE; + } + + if (cpi->redir_lid == 0) { + /* GID redirection: get PathRecord information */ + OSM_LOG(pm->log, OSM_LOG_ERROR, "ERR 4C21: " + "GID redirection not currently supported\n"); goto Exit; } @@ -1144,14 +1303,23 @@ static void osm_pc_rcv_process(void *context, void *data) p_mon_node->redir_tbl_size); goto Exit; } + p_mon_node->redir_port[port].redirection = TRUE; + p_mon_node->redir_port[port].invalid = invalid; + memcpy(&p_mon_node->redir_port[port].redir_gid, + &cpi->redir_gid, sizeof(ib_gid_t)); p_mon_node->redir_port[port].redir_lid = cpi->redir_lid; p_mon_node->redir_port[port].redir_qp = cpi->redir_qp; + p_mon_node->redir_port[port].redir_pkey = cpi->redir_pkey; + p_mon_node->redir_port[port].redir_pkey_ix = pkey_ix; cl_plock_release(pm->lock); + if (invalid) + goto Exit; + /* Finally, reissue the query to the redirected location */ status = osm_perfmgr_send_pc_mad(pm, cpi->redir_lid, cpi->redir_qp, - port, + pkey_ix, port, mad_context->perfmgr_context. mad_method, mad_context); if (status != IB_SUCCESS) @@ -1234,6 +1402,7 @@ osm_perfmgr_init(osm_perfmgr_t * const pm, osm_opensm_t *osm, pm->sweep_time_s = p_opt->perfmgr_sweep_time_s; pm->max_outstanding_queries = p_opt->perfmgr_max_outstanding_queries; pm->osm = osm; + pm->local_port = -1; status = cl_timer_init(&pm->sweep_timer, perfmgr_sweep, pm); if (status != IB_SUCCESS) From hnrose at comcast.net Thu Mar 12 13:20:08 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 12 Mar 2009 15:20:08 -0500 Subject: [ofa-general] ***SPAM*** [PATCHv2 2/2] opensm: Utilize ib_gid_is_notzero routine Message-ID: <20090312202008.GB25024@comcast.net> Signed-off-by: Hal Rosenstock --- Changes since v1: Rebased osm_sa_path_record.c diff --git a/opensm/opensm/osm_inform.c b/opensm/opensm/osm_inform.c index f0dab30..dd3f41e 100644 --- a/opensm/opensm/osm_inform.c +++ b/opensm/opensm/osm_inform.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -108,7 +109,6 @@ __match_inf_rec(IN const cl_list_item_t * const p_list_item, IN void *context) osm_infr_t *p_infr = (osm_infr_t *) p_list_item; osm_log_t *p_log = p_infr_rec->sa->p_log; cl_status_t status = CL_NOT_FOUND; - ib_gid_t all_zero_gid; OSM_LOG_ENTER(p_log); @@ -118,11 +118,8 @@ __match_inf_rec(IN const cl_list_item_t * const p_list_item, IN void *context) goto Exit; } - memset(&all_zero_gid, 0, sizeof(ib_gid_t)); - /* if inform_info.gid is not zero, ignore lid range */ - if (memcmp(&p_infr_rec->inform_record.inform_info.gid, &all_zero_gid, - sizeof(p_infr_rec->inform_record.inform_info.gid))) { + if (ib_gid_is_notzero(&p_infr_rec->inform_record.inform_info.gid)) { if (memcmp(&p_infr->inform_record.inform_info.gid, &p_infr_rec->inform_record.inform_info.gid, sizeof(p_infr->inform_record.inform_info.gid))) { diff --git a/opensm/opensm/osm_sa_informinfo.c b/opensm/opensm/osm_sa_informinfo.c index 5863753..fb81008 100644 --- a/opensm/opensm/osm_sa_informinfo.c +++ b/opensm/opensm/osm_sa_informinfo.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -95,7 +96,6 @@ __validate_ports_access_rights(IN osm_sa_t * sa, ib_net16_t lid_range_end; ib_net16_t lid; const cl_ptr_vector_t *p_tbl; - ib_gid_t zero_gid; OSM_LOG_ENTER(sa->p_log); @@ -103,9 +103,7 @@ __validate_ports_access_rights(IN osm_sa_t * sa, p_requester_physp = osm_get_physp_by_mad_addr(sa->p_log, sa->p_subn, &p_infr_rec->report_addr); - memset(&zero_gid, 0, sizeof(zero_gid)); - if (memcmp(&(p_infr_rec->inform_record.inform_info.gid), - &zero_gid, sizeof(ib_gid_t))) { + if (ib_gid_is_notzero(&p_infr_rec->inform_record.inform_info.gid)) { /* a gid is defined */ portguid = p_infr_rec->inform_record.inform_info.gid.unicast. diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index 431977d..978c1a5 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -2,7 +2,8 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. - * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. + * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -83,12 +84,6 @@ typedef struct osm_path_parms { boolean_t reversible; } osm_path_parms_t; -static const ib_gid_t zero_gid = { {0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00, - 0x00, 0x00, 0x00, 0x00}, -}; - /********************************************************************** **********************************************************************/ static inline boolean_t @@ -775,10 +770,8 @@ __osm_pr_rcv_build_pr(IN osm_sa_t * sa, p_src_physp = p_src_port->p_physp; - if (p_dgid) { - if (memcmp(p_dgid, &zero_gid, sizeof(*p_dgid))) - is_nonzero_gid = 1; - } + if (p_dgid) + is_nonzero_gid = ib_gid_is_notzero(p_dgid); if (is_nonzero_gid) p_pr->dgid = *p_dgid; From hycsw at sandia.gov Thu Mar 12 12:42:24 2009 From: hycsw at sandia.gov (Chen, Helen Y) Date: Thu, 12 Mar 2009 13:42:24 -0600 Subject: [ofa-general] Re: general Digest, Vol 26, Issue 73 Message-ID: ----- Original Message ----- From: general-bounces at lists.openfabrics.org To: general at lists.openfabrics.org Sent: Thu Mar 12 13:00:04 2009 Subject: general Digest, Vol 26, Issue 73 Send general mailing list submissions to general at lists.openfabrics.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general or, via email, send a message with subject or body 'help' to general-request at lists.openfabrics.org You can reach the person managing the list at general-owner at lists.openfabrics.org When replying, please edit your Subject line so it is more specific than "Re: Contents of general digest..." Today's Topics: 1. Re: Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC (Hal Rosenstock) 2. RE: Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC (Sean Hefty) 3. Re: Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC (Hal Rosenstock) 4. Re: [PATCH] opensm/osm_ucast_ftree.c: remove __osm_ftree prefix in static functions (Sasha Khapyorsky) 5. Re: [PATCH v2] opensm/osm_link_mgr.c initialize SMSL (Sasha Khapyorsky) 6. ***SPAM*** Re: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC (Hal Rosenstock) 7. Re: [PATCH 1/2] opensm: Add common ib_gid_is_notzero routine (Sasha Khapyorsky) 8. Re: ***SPAM*** Re: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC (Ira Weiny) 9. RE: ibdmchk: How to generate PSL and SL2VL files ? (Eitan Zahavi) ---------------------------------------------------------------------- Message: 1 Date: Thu, 12 Mar 2009 13:31:23 -0400 From: Hal Rosenstock Subject: Re: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC To: Sasha Khapyorsky Cc: Hal Rosenstock , general at lists.openfabrics.org Message-ID: Content-Type: text/plain; charset=UTF-8 On Thu, Mar 12, 2009 at 12:35 PM, Sasha Khapyorsky wrote: > On 11:55 Thu 12 Mar ?? ?? , Hal Rosenstock wrote: >> >> Since these macros exit, now need to call mad_rpc_close_port >> prior to invoking them > > Both IBERROR and IBPANIC calls exit(), it closes all opened files. This > patch looks like an overkill. It's not a matter of relying on exit for open fds but rather the allocated memory under the covers of mad_rpc_open_port so no longer can one rely on just exit and this needs to be made explicit. -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > ------------------------------ Message: 2 Date: Thu, 12 Mar 2009 10:35:58 -0700 From: "Sean Hefty" Subject: RE: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC To: "'Hal Rosenstock'" , "Sasha Khapyorsky" Cc: Hal Rosenstock , general at lists.openfabrics.org Message-ID: Content-Type: text/plain; charset="us-ascii" >It's not a matter of relying on exit for open fds but rather the >allocated memory under the covers of mad_rpc_open_port so no longer >can one rely on just exit and this needs to be made explicit. The OS should reclaim any allocated memory not freed by the app when it exits. Is this your concern? ------------------------------ Message: 3 Date: Thu, 12 Mar 2009 13:45:22 -0400 From: Hal Rosenstock Subject: Re: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC To: Sean Hefty Cc: Hal Rosenstock , general at lists.openfabrics.org Message-ID: Content-Type: text/plain; charset=UTF-8 On Thu, Mar 12, 2009 at 1:35 PM, Sean Hefty wrote: >>It's not a matter of relying on exit for open fds but rather the >>allocated memory under the covers of mad_rpc_open_port so no longer >>can one rely on just exit and this needs to be made explicit. > > The OS should reclaim any allocated memory not freed by the app when it exits. > Is this your concern? malloc'd memory ? ------------------------------ Message: 4 Date: Thu, 12 Mar 2009 19:53:03 +0200 From: Sasha Khapyorsky Subject: [ofa-general] Re: [PATCH] opensm/osm_ucast_ftree.c: remove __osm_ftree prefix in static functions To: Yevgeny Kliteynik Cc: OpenIB Message-ID: <20090312175303.GF8818 at sashak.voltaire.com> Content-Type: text/plain; charset=us-ascii On 11:30 Thu 12 Mar , Yevgeny Kliteynik wrote: > Hi Sasha, > > Removing all the __osm_ftree prefixes in static functions naming. > > Signed-off-by: Yevgeny Kliteynik Applied. Thanks. I also formatted osm_ucast_ftree.c with osm_indent. Sasha ------------------------------ Message: 5 Date: Thu, 12 Mar 2009 20:07:48 +0200 From: Sasha Khapyorsky Subject: [ofa-general] Re: [PATCH v2] opensm/osm_link_mgr.c initialize SMSL To: Line.Holen at Sun.COM Cc: general at lists.openfabrics.org Message-ID: <20090312180741.GG8818 at sashak.voltaire.com> Content-Type: text/plain; charset=us-ascii On 10:37 Tue 10 Mar , Line.Holen at Sun.COM wrote: > The SMSL is currently not initialized. Whatever is written in PortInfo > at power up is kept. > > The lash routing algorithm makes active use of SL to avoid deadlock in > the fabric. The endnodes needs to be instructed to use the SL calculated > for the endnode-to-SMnode path for SA traffic. If not, the SA traffic > can cause deadlock. > > This change makes sure that the SMSL is properly initialized on all > ports. If flash routing is used, the SMSL is set up based on routing > calculations. For other routing algorithms the default SMSL is used. > > > Signed-off-by: Line Holen Applied. Thanks. Sasha ------------------------------ Message: 6 Date: Thu, 12 Mar 2009 14:05:09 -0400 From: Hal Rosenstock Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC To: Sean Hefty Cc: Hal Rosenstock , general at lists.openfabrics.org Message-ID: Content-Type: text/plain; charset=UTF-8 On Thu, Mar 12, 2009 at 1:45 PM, Hal Rosenstock wrote: > On Thu, Mar 12, 2009 at 1:35 PM, Sean Hefty wrote: >>>It's not a matter of relying on exit for open fds but rather the >>>allocated memory under the covers of mad_rpc_open_port so no longer >>>can one rely on just exit and this needs to be made explicit. >> >> The OS should reclaim any allocated memory not freed by the app when it exits. >> Is this your concern? > > malloc'd memory ? Many OSs do but not all. Should we rely on this ? Also, don't other userspace programs attempt to properly cleanup after themselves rather than rely on OS cleanup behavior ? ------------------------------ Message: 7 Date: Thu, 12 Mar 2009 20:19:51 +0200 From: Sasha Khapyorsky Subject: [ofa-general] Re: [PATCH 1/2] opensm: Add common ib_gid_is_notzero routine To: Hal Rosenstock Cc: general at lists.openfabrics.org Message-ID: <20090312181951.GH8818 at sashak.voltaire.com> Content-Type: text/plain; charset=us-ascii On 14:22 Tue 10 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock > --- > diff --git a/opensm/include/iba/ib_types.h b/opensm/include/iba/ib_types.h > index e7cbb79..bfe8665 100644 > --- a/opensm/include/iba/ib_types.h > +++ b/opensm/include/iba/ib_types.h > @@ -2338,6 +2338,40 @@ ib_gid_get_guid(IN const ib_gid_t * const p_gid) > * ib_gid_t > *********/ > > +OSM_EXPORT const ib_gid_t ib_zero_gid; > + > +/****f* IBA Base: Types/ib_gid_is_notzero > +* NAME > +* ib_gid_is_notzero > +* > +* DESCRIPTION > +* Returns a boolean indicating whether or not the GID is zero. > +* > +* SYNOPSIS > +*/ > +static inline boolean_t OSM_API > +ib_gid_is_notzero(IN const ib_gid_t * const p_gid) > +{ > + if (memcmp(p_gid, &ib_zero_gid, sizeof(*p_gid))) > + return TRUE; > + return FALSE; > +} [snip...] > diff --git a/opensm/opensm/osm_helper.c b/opensm/opensm/osm_helper.c > index c56f5b2..b40ba0c 100644 > --- a/opensm/opensm/osm_helper.c > +++ b/opensm/opensm/osm_helper.c > @@ -535,6 +535,8 @@ const char *ib_get_trap_str(ib_net16_t trap_num) > return "Unknown"; > } > > +const ib_gid_t ib_zero_gid = {}; > + This introduces dependency against libopensm for ib_types.h users. I would suggest to put all this stuff to osm_helper. Sasha ------------------------------ Message: 8 Date: Thu, 12 Mar 2009 11:23:07 -0700 From: Ira Weiny Subject: Re: ***SPAM*** Re: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC To: Hal Rosenstock Cc: Hal Rosenstock , general at lists.openfabrics.org Message-ID: <20090312112307.972a44ec.weiny2 at llnl.gov> Content-Type: text/plain; charset=US-ASCII On Thu, 12 Mar 2009 14:05:09 -0400 Hal Rosenstock wrote: > On Thu, Mar 12, 2009 at 1:45 PM, Hal Rosenstock > wrote: > > On Thu, Mar 12, 2009 at 1:35 PM, Sean Hefty wrote: > >>>It's not a matter of relying on exit for open fds but rather the > >>>allocated memory under the covers of mad_rpc_open_port so no longer > >>>can one rely on just exit and this needs to be made explicit. > >> > >> The OS should reclaim any allocated memory not freed by the app when it exits. > >> Is this your concern? > > > > malloc'd memory ? > > Many OSs do but not all. Should we rely on this ? Also, don't other > userspace programs attempt to properly cleanup after themselves rather > than rely on OS cleanup behavior ? Strictly speaking I think Hal is right on this one. I should have made the change myself was lazy. Another more tangible reason for this is we may change the behavior of the macros in the future. IBERROR recently fooled Sasha into thinking execution continued. IBPANIC is perhaps a bit more clear that it will exit. Ira > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http:// lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http:// openib.org/mailman/listinfo/openib-general > -- Ira Weiny Math Programer/Computer Scientist Larence Livermore National Lab weiny2 at llnl.gov ------------------------------ Message: 9 Date: Thu, 12 Mar 2009 20:06:17 +0200 From: "Eitan Zahavi" Subject: [ofa-general] RE: ibdmchk: How to generate PSL and SL2VL files ? To: "Keshetti Mahesh" Cc: openIB Message-ID: <40FA0A8088E8A441973D37502F00933E0140FCFE at mtlexch01.mtl.com> Content-Type: text/plain; charset="us-ascii" Hi Mahesh, The help message is erroneous: First these file are not mandatory. Second OpenSM does not generate them. The files can be generated by performing SA PathRecord queries. We do not automation for that yet. So it seems we need to perform 2 things: 1. Fix the help / man 2. Write some SA client code for obtaining the data. Thanks for pointing this up. Eitan Zahavi Senior Engineering Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL > -----Original Message----- > From: Keshetti Mahesh [mailto:keshetti.mahesh at gmail.com] > Sent: Thursday, March 12, 2009 8:27 AM > To: Eitan Zahavi > Cc: openIB > Subject: Re: ibdmchk: How to generate PSL and SL2VL files ? > > On Thu, Mar 12, 2009 at 11:52 AM, Keshetti Mahesh > wrote: > > Hi, > > > > Current implementation of 'ibdmchk' in cluster verification mode > > requires path<->SL(PSL) and SL2VL files as input. And the help(-h) > > message says that these files can be generated by OpenSM > with '-D 0x43'. > > > > But I am not able to generate any such file with OpenSM. And there > > isn't any information related to these files in man page as well as > > help(-h). Can anyone please tell me how to generate > path<->SL and SL2VL files using OpenSM? > > > > I am using OpenSM of OFED-1.4 distribution. > > -Mahesh > ------------------------------ _______________________________________________ general mailing list general at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general End of general Digest, Vol 26, Issue 73 *************************************** From weiny2 at llnl.gov Thu Mar 12 13:39:49 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Thu, 12 Mar 2009 13:39:49 -0700 Subject: [ofa-general] OpenSM build now requires lex (flex) and yacc (bison)? Message-ID: <20090312133949.2fe26c07.weiny2@llnl.gov> Doug, Sasha, We are rolling out internal builds of OpenSM 3.2.5 and found that we needed the following patch to build under the mock build system. As far as I can tell osm_qos_parser.l has been around for a long time. So I don't know why this is a new requirement. Doug, did you have to do something to build 3.2.2 in RedHat's build system? Ira >From a65afe305864049b00642759dc2fb8e4d0499c66 Mon Sep 17 00:00:00 2001 From: Tim Meier Date: Thu, 12 Mar 2009 07:54:27 -0700 Subject: [PATCH] added yacc requirement to build dependency Signed-off-by: Tim Meier --- opensm/opensm.spec.in | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/opensm/opensm.spec.in b/opensm/opensm.spec.in index 7b82faf..3f02235 100644 --- a/opensm/opensm.spec.in +++ b/opensm/opensm.spec.in @@ -30,7 +30,7 @@ Group: System Environment/Daemons URL: http://openfabrics.org/ Source: http://www.openfabrics.org/downloads/management/@TARBALL@ BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) -BuildRequires: libibumad-devel, libtool +BuildRequires: bison, flex, libibumad-devel, libtool Requires: %{name}-libs = %{version}-%{release}, logrotate Requires(post): /sbin/service, /sbin/chkconfig Requires(preun): /sbin/chkconfig, /sbin/service -- 1.5.4.5 From jgunthorpe at obsidianresearch.com Thu Mar 12 15:50:29 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Thu, 12 Mar 2009 16:50:29 -0600 Subject: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC In-Reply-To: <20090312112307.972a44ec.weiny2@llnl.gov> References: <20090312165530.GA10229@comcast.net> <20090312163527.GA8818@sashak.voltaire.com> <20090312112307.972a44ec.weiny2@llnl.gov> Message-ID: <20090312225029.GD31138@obsidianresearch.com> On Thu, Mar 12, 2009 at 11:23:07AM -0700, Ira Weiny wrote: > > >>>It's not a matter of relying on exit for open fds but rather the > > >>>allocated memory under the covers of mad_rpc_open_port so no longer > > >>>can one rely on just exit and this needs to be made explicit. > > >> > > >> The OS should reclaim any allocated memory not freed by the app when it exits. > > >> Is this your concern? > > > > > > malloc'd memory ? > > > > Many OSs do but not all. Should we rely on this ? Also, don't other > > userspace programs attempt to properly cleanup after themselves rather > > than rely on OS cleanup behavior ? > Strictly speaking I think Hal is right on this one. I should have > made the change myself was lazy. I've never heard of or used an OS that has processes (ie exit) and didn't reclaim FDs and memory. The closest I've ever used was classic Windows 3.1 vintage where you had processes, but some GUI resources could leak if not freed. malloc didn't leak. Heck, even DOS reclaimed straight malloc'd memory. POSIX environments reclaim virtually everything on exit, though there are some APIs for shared memory and some types of non POSIX semaphores that can linger due to their multi-process nature. But these are rare and need special handling anyhow.. Certainly attempting to unwind malloc, fds, etc at exit on any sort of POSIX platform is needless busy work. When you care about stuff like this is if you are making functions that don't have the luxury of calling exit/abort/etc, like in the kernel or in some kind of RTOS environment - but in that situation it is exit that is forbidden and proper resource clean up and error code returning is just a consequence of getting rid of exit. Jason From kliteyn at dev.mellanox.co.il Thu Mar 12 16:04:41 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Fri, 13 Mar 2009 01:04:41 +0200 Subject: [ofa-general] [PATCH v2] opensm/osm_ucast_ftree.c: some refactoring Message-ID: <49B99509.9030000@dev.mellanox.co.il> Hi Sasha, Some code refactoring: setting entry in the lid matrices should be done per port group. V2 - patch reworked. Signed-off-by: Yevgeny Kliteynik --- opensm/opensm/osm_ucast_ftree.c | 67 ++++++++++++++++++--------------------- 1 files changed, 31 insertions(+), 36 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index e841368..941fc79 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -698,6 +698,28 @@ static inline cl_status_t sw_set_hops(IN ftree_sw_t * p_sw, IN uint16_t lid_ho, return osm_switch_set_hops(p_sw->p_osm_sw, lid_ho, port_num, hops); } +/***************************************************/ + +static int +set_hops_on_remote_sw(IN ftree_port_group_t * p_group, + IN ib_net16_t target_lid, + IN uint8_t hops) +{ + ftree_port_t * p_port; + uint8_t i, ports_num; + ftree_sw_t * p_remote_sw = p_group->remote_hca_or_sw.p_sw; + + CL_ASSERT(p_group->remote_node_type == IB_NODE_TYPE_SWITCH); + ports_num = (uint8_t) cl_ptr_vector_get_size(&p_group->ports); + for (i = 0; i < ports_num; i++) { + cl_ptr_vector_at(&p_group->ports, i, (void *)&p_port); + if (sw_set_hops(p_remote_sw, cl_ntoh16(target_lid), + p_port->remote_port_num, hops)); + return -1; + } + return 0; +} + /*************************************************** ** ** ftree_hca_t functions @@ -1988,20 +2010,10 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, /* On the remote switch that is pointed by the p_group, set hops for ALL the ports in the remote group. */ - for (j = 0; j < ports_num; j++) { - cl_ptr_vector_at(&p_group->ports, j, - (void *)&p_port); - - sw_set_hops(p_remote_sw, - cl_ntoh16(target_lid), - p_port->remote_port_num, - ((target_rank - - highest_rank_in_route) + - (p_remote_sw->rank - - highest_rank_in_route) - + reverse_hops * 2)); - } - + set_hops_on_remote_sw(p_group, target_lid, + ((target_rank - highest_rank_in_route) + + (p_remote_sw->rank - highest_rank_in_route) + + reverse_hops * 2)); } /* The number of upgoing routes is tracked in the @@ -2213,18 +2225,9 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* On the remote switch that is pointed by the min_group, set hops for ALL the ports in the remote group. */ - ports_num = - (uint16_t) cl_ptr_vector_get_size(&p_min_group-> - ports); - for (j = 0; j < ports_num; j++) { - cl_ptr_vector_at(&p_min_group->ports, j, - (void *)&p_port); - sw_set_hops(p_remote_sw, - cl_ntoh16(target_lid), - p_port->remote_port_num, - target_rank - p_remote_sw->rank + - 2 * reverse_hops); - } + set_hops_on_remote_sw(p_min_group, target_lid, + target_rank - p_remote_sw->rank + + 2 * reverse_hops); } /* Recursion step: @@ -2305,16 +2308,8 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* On the remote switch that is pointed by the p_group, set hops for ALL the ports in the remote group. */ - ports_num = (uint16_t) cl_ptr_vector_get_size(&p_group->ports); - for (j = 0; j < ports_num; j++) { - cl_ptr_vector_at(&p_group->ports, j, (void *)&p_port); - - sw_set_hops(p_remote_sw, - cl_ntoh16(target_lid), - p_port->remote_port_num, - target_rank - p_remote_sw->rank + - 2 * reverse_hops); - } + set_hops_on_remote_sw(p_group, target_lid, + target_rank - p_remote_sw->rank + 2 * reverse_hops); /* Recursion step: Assign downgoing ports by stepping up, starting on REMOTE switch. */ -- 1.5.1.4 From arlin.r.davis at intel.com Thu Mar 12 16:20:26 2009 From: arlin.r.davis at intel.com (Arlin Davis) Date: Thu, 12 Mar 2009 16:20:26 -0700 Subject: [ofa-general] [PATCH 1/2] uDAPL Build: spec files missing Requires(post) statements for sed/coreutils Message-ID: <000001c9a369$20926b50$ce97070a@amr.corp.intel.com> needed for anaconda install Signed-off-by: Arlin Davis --- dapl.spec.in | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/dapl.spec.in b/dapl.spec.in index 3197c1b..1192d0a 100644 --- a/dapl.spec.in +++ b/dapl.spec.in @@ -43,6 +43,8 @@ Source: http://www.openfabrics.org/downloads/%{name}/%{name}-%{version}.tar.gz BuildRoot: %(mktemp -ud %{_tmppath}/%{name}-%{version}-%{release}-XXXXXX) Requires(post): /sbin/ldconfig Requires(postun): /sbin/ldconfig +Requires(post): sed +Requires(post): coreutils %description Along with the OpenFabrics kernel drivers, libdat and libdapl provides a userspace -- 1.5.2.5 From arlin.r.davis at intel.com Thu Mar 12 16:20:31 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Thu, 12 Mar 2009 16:20:31 -0700 Subject: [ofa-general] [PATCH 2/2] uDAPL Build: missing new linux/osd include file in EXTRA_DIST Message-ID: Add dapl/openib_scm/linux/openib_osd.h to EXTRA_DIST Signed-off-by: Arlin Davis --- Makefile.am | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/Makefile.am b/Makefile.am index 5044e36..08a8cce 100755 --- a/Makefile.am +++ b/Makefile.am @@ -364,6 +364,7 @@ EXTRA_DIST = dat/common/dat_dictionary.h \ dapl/openib_cma/dapl_ib_util.h \ dapl/openib_scm/dapl_ib_dto.h \ dapl/openib_scm/dapl_ib_util.h \ + dapl/openib_scm/linux/openib_osd.h \ dat/udat/libdat2.map \ dapl/udapl/libdaplofa.map \ dapl/udapl/libdaploscm.map \ -- 1.5.2.5 From sashak at voltaire.com Fri Mar 13 00:58:29 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 13 Mar 2009 09:58:29 +0200 Subject: [ofa-general] Re: [PATCH v2] opensm/osm_ucast_ftree.c: some refactoring In-Reply-To: <49B99509.9030000@dev.mellanox.co.il> References: <49B99509.9030000@dev.mellanox.co.il> Message-ID: <20090313075829.GJ8818@sashak.voltaire.com> Hi Yevgeny, On 01:04 Fri 13 Mar , Yevgeny Kliteynik wrote: > > +/***************************************************/ > + > +static int > +set_hops_on_remote_sw(IN ftree_port_group_t * p_group, > + IN ib_net16_t target_lid, > + IN uint8_t hops) > +{ > + ftree_port_t * p_port; > + uint8_t i, ports_num; > + ftree_sw_t * p_remote_sw = p_group->remote_hca_or_sw.p_sw; > + > + CL_ASSERT(p_group->remote_node_type == IB_NODE_TYPE_SWITCH); > + ports_num = (uint8_t) cl_ptr_vector_get_size(&p_group->ports); > + for (i = 0; i < ports_num; i++) { > + cl_ptr_vector_at(&p_group->ports, i, (void *)&p_port); > + if (sw_set_hops(p_remote_sw, cl_ntoh16(target_lid), > + p_port->remote_port_num, hops)); Should ';' be here at end of if()? Sasha > + return -1; > + } > + return 0; > +} > + > /*************************************************** > ** > ** ftree_hca_t functions From vlad at lists.openfabrics.org Fri Mar 13 03:18:18 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Fri, 13 Mar 2009 03:18:18 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090313-0200 daily build status Message-ID: <20090313101818.D5883E61102@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From hal.rosenstock at gmail.com Fri Mar 13 04:30:54 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 13 Mar 2009 07:30:54 -0400 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] infiniband-diags/perfquery.8: Update man page for PortXmit/RcvDataSL In-Reply-To: <20090312163013.GZ8818@sashak.voltaire.com> References: <20090310205023.GA1791@comcast.net> <20090312143115.GM8818@sashak.voltaire.com> <20090312163013.GZ8818@sashak.voltaire.com> Message-ID: On Thu, Mar 12, 2009 at 12:30 PM, Sasha Khapyorsky wrote: > On 11:59 Thu 12 Mar     , Hal Rosenstock wrote: >> >> I did (and do in general). Not sure why this one was broken. > > It was was generated against some version where -e was changed to -x. I must have skipped a step here; Sorry. -- Hal > Sasha > From hal.rosenstock at gmail.com Fri Mar 13 04:31:24 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 13 Mar 2009 07:31:24 -0400 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC In-Reply-To: <20090312225029.GD31138@obsidianresearch.com> References: <20090312165530.GA10229@comcast.net> <20090312163527.GA8818@sashak.voltaire.com> <20090312112307.972a44ec.weiny2@llnl.gov> <20090312225029.GD31138@obsidianresearch.com> Message-ID: On Thu, Mar 12, 2009 at 6:50 PM, Jason Gunthorpe wrote: > I've never heard of or used an OS that has processes (ie exit) and > didn't reclaim FDs and memory. The closest I've ever used was classic > Windows 3.1 vintage where you had processes, but some GUI resources > could leak if not freed. malloc didn't leak. Heck, even DOS reclaimed > straight malloc'd memory. There are a number of "unix" like RTOSs which support abort and exit without resource tracking/reclamation. > POSIX environments reclaim virtually everything on exit, though there > are some APIs for shared memory and some types of non POSIX semaphores > that can linger due to their multi-process nature. But these are rare > and need special handling anyhow.. > > Certainly attempting to unwind malloc, fds, etc at exit on any sort of > POSIX platform is needless busy work. > > When you care about stuff like this is if you are making functions > that don't have the luxury of calling exit/abort/etc, like in the > kernel or in some kind of RTOS environment - but in that situation it > is exit that is forbidden and proper resource clean up and error code > returning is just a consequence of getting rid of exit. That's not true in a number of RTOSs I've worked with in the past (vxWorks is the first one which comes to mind). -- Hal > Jason > From bart.vanassche at gmail.com Fri Mar 13 05:00:26 2009 From: bart.vanassche at gmail.com (Bart Van Assche) Date: Fri, 13 Mar 2009 13:00:26 +0100 Subject: ***SPAM*** Re: ***SPAM*** Re: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC In-Reply-To: References: <20090312165530.GA10229@comcast.net> <20090312163527.GA8818@sashak.voltaire.com> Message-ID: On Thu, Mar 12, 2009 at 7:05 PM, Hal Rosenstock wrote: > On Thu, Mar 12, 2009 at 1:45 PM, Hal Rosenstock > wrote: >> On Thu, Mar 12, 2009 at 1:35 PM, Sean Hefty wrote: >>>>It's not a matter of relying on exit for open fds but rather the >>>>allocated memory under the covers of mad_rpc_open_port so no longer >>>>can one rely on just exit and this needs to be made explicit. >>> >>> The OS should reclaim any allocated memory not freed by the app when it exits. >>> Is this your concern? >> >> malloc'd memory ? > > Many OSs do but not all. Should we rely on this ? Also, don't other > userspace programs attempt to properly cleanup after themselves rather > than rely on OS cleanup behavior ? Letting an application deallocate all allocated resources before it exits makes it easier to analyze these applications with run-time analysis tools like Valgrind. Valgrind is able to perform memory leak checking and file descriptor leak detection. Bart. From jgunthorpe at obsidianresearch.com Fri Mar 13 09:37:54 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Fri, 13 Mar 2009 10:37:54 -0600 Subject: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC In-Reply-To: References: <20090312165530.GA10229@comcast.net> <20090312163527.GA8818@sashak.voltaire.com> <20090312112307.972a44ec.weiny2@llnl.gov> <20090312225029.GD31138@obsidianresearch.com> Message-ID: <20090313163754.GA23358@obsidianresearch.com> On Fri, Mar 13, 2009 at 07:31:24AM -0400, Hal Rosenstock wrote: > > When you care about stuff like this is if you are making functions > > that don't have the luxury of calling exit/abort/etc, like in the > > kernel or in some kind of RTOS environment - but in that situation it > > is exit that is forbidden and proper resource clean up and error code > > returning is just a consequence of getting rid of exit. > > That's not true in a number of RTOSs I've worked with in the past > (vxWorks is the first one which comes to mind). vxWorks doesn't have processes, that they have a library call called 'exit' that happens to work like thread_exit doesn't matter one bit, you still can't call it and you still have to do proper resource clean up and error unwind by hand. It is just plain impossible to call exit/assert/abort from a library and get proper resource clean up in an environment without processes. Worrying about a few extra malloc the library may have made while a call is running is pointless because you are ignoring all the mallocs the calling program has done that won't get unwound. Either free absolutely *everything* by guaranteeing that every library call returns back to the caller - or don't: call exit and require your library run in an environment with processes. Mixing the two approaches is pointless and confusing. Jason From jgunthorpe at obsidianresearch.com Fri Mar 13 09:40:00 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Fri, 13 Mar 2009 10:40:00 -0600 Subject: ***SPAM*** Re: ***SPAM*** Re: [ofa-general] Re: [PATCH] infiniband-diags: Fix memory leaks on IBERROR and IBPANIC In-Reply-To: References: <20090312165530.GA10229@comcast.net> <20090312163527.GA8818@sashak.voltaire.com> Message-ID: <20090313164000.GB23358@obsidianresearch.com> On Fri, Mar 13, 2009 at 01:00:26PM +0100, Bart Van Assche wrote: > Letting an application deallocate all allocated resources before it > exits makes it easier to analyze these applications with run-time > analysis tools like Valgrind. Valgrind is able to perform memory leak > checking and file descriptor leak detection. This is about what to do on an error condition in the program, it doesn't make much sense to do a leak detection analysis with valgrind on error paths that result in process exit. Jason From hnrose at comcast.net Fri Mar 13 13:36:20 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Fri, 13 Mar 2009 15:36:20 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] libibmad: Add PortSelect and CounterSelect fields for PortXmit/RcvDataSL attributes Message-ID: <20090313203620.GA6173@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h index b8290a7..96bb482 100644 --- a/libibmad/include/infiniband/mad.h +++ b/libibmad/include/infiniband/mad.h @@ -564,7 +564,9 @@ enum MAD_FIELDS { IB_CPI_TRAP_QKEY_F, IB_PC_XMT_DATA_SL_FIRST_F, - IB_PC_XMT_DATA_SL0_F = IB_PC_XMT_DATA_SL_FIRST_F, + IB_PC_XMT_DATA_SL_PORT_SELECT_F = IB_PC_XMT_DATA_SL_FIRST_F, + IB_PC_XMT_DATA_SL_COUNTER_SELECT_F, + IB_PC_XMT_DATA_SL0_F, IB_PC_XMT_DATA_SL1_F, IB_PC_XMT_DATA_SL2_F, IB_PC_XMT_DATA_SL3_F, @@ -583,7 +585,9 @@ enum MAD_FIELDS { IB_PC_XMT_DATA_SL_LAST_F, IB_PC_RCV_DATA_SL_FIRST_F, - IB_PC_RCV_DATA_SL0_F = IB_PC_RCV_DATA_SL_FIRST_F, + IB_PC_RCV_DATA_SL_PORT_SELECT_F = IB_PC_RCV_DATA_SL_FIRST_F, + IB_PC_RCV_DATA_SL_COUNTER_SELECT_F, + IB_PC_RCV_DATA_SL0_F, IB_PC_RCV_DATA_SL1_F, IB_PC_RCV_DATA_SL2_F, IB_PC_RCV_DATA_SL3_F, diff --git a/libibmad/src/fields.c b/libibmad/src/fields.c index 19c8fc1..ff94c45 100644 --- a/libibmad/src/fields.c +++ b/libibmad/src/fields.c @@ -402,6 +402,8 @@ static const ib_field_t ib_mad_f[] = { {BITSOFFS(520, 24), "TrapQP", mad_dump_hex}, {544, 32, "TrapQKey", mad_dump_hex}, + {BITSOFFS(8, 8), "PortSelect", mad_dump_uint}, + {BITSOFFS(16, 16), "CounterSelect", mad_dump_hex}, {32, 32, "XmtDataSL0", mad_dump_uint}, {64, 32, "XmtDataSL1", mad_dump_uint}, {96, 32, "XmtDataSL2", mad_dump_uint}, @@ -420,6 +422,8 @@ static const ib_field_t ib_mad_f[] = { {512, 32, "XmtDataSL15", mad_dump_uint}, {0, 0}, /* IB_PC_XMT_DATA_SL_LAST_F */ + {BITSOFFS(8, 8), "PortSelect", mad_dump_uint}, + {BITSOFFS(16, 16), "CounterSelect", mad_dump_hex}, {32, 32, "RcvDataSL0", mad_dump_uint}, {64, 32, "RcvDataSL1", mad_dump_uint}, {96, 32, "RcvDataSL2", mad_dump_uint}, From hnrose at comcast.net Fri Mar 13 13:37:18 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Fri, 13 Mar 2009 15:37:18 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] infiniband-diags/perfquery.c: Label PortXmit/RcvDataSL counters in headings Message-ID: <20090313203718.GB6173@comcast.net> Also, some cosmetic formatting changes Signed-off-by: Hal Rosenstock --- diff --git a/infiniband-diags/src/perfquery.c b/infiniband-diags/src/perfquery.c index d35e995..b0bda49 100644 --- a/infiniband-diags/src/perfquery.c +++ b/infiniband-diags/src/perfquery.c @@ -331,9 +331,9 @@ void xmt_sl_query(ib_portid_t *portid, int port, int mask) IBERROR("perfslquery"); mad_dump_perfcounters_xmt_sl(buf, sizeof buf, pc, sizeof pc); - printf("# Port counters: %s port %d\n%s", portid2str(portid), port, buf); + printf("# PortXmitDataSL counters: %s port %d\n%s", portid2str(portid), port, buf); - if(reset) + if (reset) if (!performance_reset_via(pc, portid, port, mask, ibd_timeout, IB_GSI_PORT_XMIT_DATA_SL, srcport)) IBERROR("perfslreset"); @@ -355,9 +355,9 @@ void rcv_sl_query(ib_portid_t *portid, int port, int mask) IBERROR("perfslquery"); mad_dump_perfcounters_rcv_sl(buf, sizeof buf, pc, sizeof pc); - printf("# Port counters: %s port %d\n%s", portid2str(portid), port, buf); + printf("# PortRcvDataSL counters: %s port %d\n%s", portid2str(portid), port, buf); - if(reset) + if (reset) if (!performance_reset_via(pc, portid, port, mask, ibd_timeout, IB_GSI_PORT_RCV_DATA_SL, srcport)) IBERROR("perfslreset"); From arlin.r.davis at intel.com Fri Mar 13 15:30:11 2009 From: arlin.r.davis at intel.com (Arlin Davis) Date: Fri, 13 Mar 2009 15:30:11 -0700 Subject: [ofa-general] [PATCH 1/1] uDAPL: scm provider, remove query gid/lid from connection setup phase to the open Message-ID: <000001c9a42b$46cc1a60$ce97070a@amr.corp.intel.com> move lid/gid queries from the connection setup phase and put them in the open call to avoid overhead of more fd's during connections. No need to query during connection setup since uDAPL binds to specific hca/ports via dat_ia_open. Signed-off-by: Arlin Davis --- dapl/openib_scm/dapl_ib_cm.c | 60 +++------------------------------------ dapl/openib_scm/dapl_ib_util.c | 32 +++++++++++++++++++++ dapl/openib_scm/dapl_ib_util.h | 4 ++- 3 files changed, 40 insertions(+), 56 deletions(-) diff --git a/dapl/openib_scm/dapl_ib_cm.c b/dapl/openib_scm/dapl_ib_cm.c index 9a15e42..9defc42 100644 --- a/dapl/openib_scm/dapl_ib_cm.c +++ b/dapl/openib_scm/dapl_ib_cm.c @@ -285,16 +285,6 @@ static void dapli_cm_queue(struct ib_cm_handle *cm_ptr) strerror(errno)); } -static uint16_t dapli_get_lid(IN struct ibv_context *ctx, IN uint8_t port) -{ - struct ibv_port_attr port_attr; - - if(ibv_query_port(ctx, port,&port_attr)) - return(0xffff); - else - return(port_attr.lid); -} - /* * ACTIVE/PASSIVE: called from CR thread or consumer via ep_disconnect */ @@ -468,28 +458,9 @@ dapli_socket_connect(DAPL_EP *ep_ptr, cm_ptr->dst.qp_type = htons(ep_ptr->qp_handle->qp_type); #endif cm_ptr->dst.port = htons(ia_ptr->hca_ptr->port_num); - cm_ptr->dst.lid = - htons(dapli_get_lid(ia_ptr->hca_ptr->ib_hca_handle, - (uint8_t)ia_ptr->hca_ptr->port_num)); - if (cm_ptr->dst.lid == 0xffff) { - dapl_log(DAPL_DBG_TYPE_ERR, - " CONNECT: query LID ERR %s -> %s\n", - strerror(errno), - inet_ntoa(((struct sockaddr_in *)r_addr)->sin_addr)); - goto bail; - } - - /* in network order */ - if (ibv_query_gid(ia_ptr->hca_ptr->ib_hca_handle, - (uint8_t)ia_ptr->hca_ptr->port_num, - 0, &cm_ptr->dst.gid)) { - dapl_log(DAPL_DBG_TYPE_ERR, - " CONNECT: query GID ERR %s -> %s\n", - strerror(errno), - inet_ntoa(((struct sockaddr_in *)r_addr)->sin_addr)); - goto bail; - } - + cm_ptr->dst.lid = ia_ptr->hca_ptr->ib_trans.lid; + cm_ptr->dst.gid = ia_ptr->hca_ptr->ib_trans.gid; + /* save references */ cm_ptr->hca = ia_ptr->hca_ptr; cm_ptr->ep = ep_ptr; @@ -958,29 +929,8 @@ dapli_socket_accept_usr(DAPL_EP *ep_ptr, local.qpn = htonl(ep_ptr->qp_handle->qp_num); local.qp_type = htons(ep_ptr->qp_handle->qp_type); local.port = htons(ia_ptr->hca_ptr->port_num); - local.lid = htons(dapli_get_lid(ia_ptr->hca_ptr->ib_hca_handle, - (uint8_t)ia_ptr->hca_ptr->port_num)); - if (local.lid == 0xffff) { - dapl_log(DAPL_DBG_TYPE_ERR, - " ACCEPT_USR: query LID ERR %s -> %s\n", - strerror(errno), - inet_ntoa(((struct sockaddr_in *) - &cm_ptr->dst.ia_address)->sin_addr)); - goto bail; - } - - /* in network order */ - if (ibv_query_gid(ia_ptr->hca_ptr->ib_hca_handle, - (uint8_t)ia_ptr->hca_ptr->port_num, - 0, &local.gid)) { - dapl_log(DAPL_DBG_TYPE_ERR, - " ACCEPT_USR: query GID ERR %s -> %s\n", - strerror(errno), - inet_ntoa(((struct sockaddr_in *) - &cm_ptr->dst.ia_address)->sin_addr)); - goto bail; - } - + local.lid = ia_ptr->hca_ptr->ib_trans.lid; + local.gid = ia_ptr->hca_ptr->ib_trans.gid; local.ia_address = ia_ptr->hca_ptr->hca_address; local.p_size = htonl(p_size); iov[0].iov_base = (void *) &local; diff --git a/dapl/openib_scm/dapl_ib_util.c b/dapl/openib_scm/dapl_ib_util.c index fdbae3c..391a0e5 100644 --- a/dapl/openib_scm/dapl_ib_util.c +++ b/dapl/openib_scm/dapl_ib_util.c @@ -196,6 +196,7 @@ DAT_RETURN dapls_ib_open_hca ( IN DAPL_HCA *hca_ptr) { struct ibv_device **dev_list; + struct ibv_port_attr port_attr; int i; DAT_RETURN dat_status = DAT_SUCCESS; @@ -238,6 +239,30 @@ found: goto err; } + /* get lid for this hca-port, network order */ + if(ibv_query_port(hca_ptr->ib_hca_handle, + (uint8_t)hca_ptr->port_num, + &port_attr)) { + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: get lid ERR for %s, err=%s\n", + ibv_get_device_name(hca_ptr->ib_trans.ib_dev), + strerror(errno)); + goto err; + } else { + hca_ptr->ib_trans.lid = htons(port_attr.lid); + } + + /* get gid for this hca-port, network order */ + if (ibv_query_gid(hca_ptr->ib_hca_handle, + (uint8_t)hca_ptr->port_num, + 0, &hca_ptr->ib_trans.gid)) { + dapl_log(DAPL_DBG_TYPE_ERR, + " open_hca: query GID ERR for %s, err=%s\n", + ibv_get_device_name(hca_ptr->ib_trans.ib_dev), + strerror(errno)); + goto err; + } + /* set RC tunables via enviroment or default */ hca_ptr->ib_trans.max_inline_send = dapl_os_get_env_val("DAPL_MAX_INLINE", INLINE_SEND_DEFAULT); @@ -325,6 +350,13 @@ found: hca_ptr->port_num, inet_ntoa(((struct sockaddr_in *) &hca_ptr->hca_address)->sin_addr)); + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, + " open_hca: LID 0x%x GID Subnet 0x"F64x" ID 0x"F64x"\n", + ntohs(hca_ptr->ib_trans.lid), + (unsigned long long) + htonll(hca_ptr->ib_trans.gid.global.subnet_prefix), + (unsigned long long) + htonll(hca_ptr->ib_trans.gid.global.interface_id)); ibv_free_device_list(dev_list); return dat_status; diff --git a/dapl/openib_scm/dapl_ib_util.h b/dapl/openib_scm/dapl_ib_util.h index fd1c24e..7c3251a 100644 --- a/dapl/openib_scm/dapl_ib_util.h +++ b/dapl/openib_scm/dapl_ib_util.h @@ -277,12 +277,13 @@ typedef enum /* ib_hca_transport_t, specific to this implementation */ typedef struct _ib_hca_transport { + union ibv_gid gid; struct ibv_device *ib_dev; ib_cq_handle_t ib_cq_empty; DAPL_OS_LOCK cq_lock; int max_inline_send; ib_thread_state_t cq_state; - DAPL_OS_THREAD cq_thread; + DAPL_OS_THREAD cq_thread; struct ibv_comp_channel *ib_cq; int cr_state; DAPL_OS_THREAD thread; @@ -293,6 +294,7 @@ typedef struct _ib_hca_transport ib_async_cq_handler_t async_cq_error; ib_async_dto_handler_t async_cq; ib_async_qp_handler_t async_qp_error; + uint16_t lid; uint8_t ack_timer; uint8_t ack_retry; uint8_t rnr_timer; -- 1.5.2.5 From kliteyn at dev.mellanox.co.il Sat Mar 14 02:19:57 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Sat, 14 Mar 2009 11:19:57 +0200 Subject: [ofa-general] Re: [PATCH v2] opensm/osm_ucast_ftree.c: some refactoring In-Reply-To: <20090313075829.GJ8818@sashak.voltaire.com> References: <49B99509.9030000@dev.mellanox.co.il> <20090313075829.GJ8818@sashak.voltaire.com> Message-ID: <49BB76BD.7020107@dev.mellanox.co.il> Hi Sasha, Sasha Khapyorsky wrote: > Hi Yevgeny, > > On 01:04 Fri 13 Mar , Yevgeny Kliteynik wrote: >> +/***************************************************/ >> + >> +static int >> +set_hops_on_remote_sw(IN ftree_port_group_t * p_group, >> + IN ib_net16_t target_lid, >> + IN uint8_t hops) >> +{ >> + ftree_port_t * p_port; >> + uint8_t i, ports_num; >> + ftree_sw_t * p_remote_sw = p_group->remote_hca_or_sw.p_sw; >> + >> + CL_ASSERT(p_group->remote_node_type == IB_NODE_TYPE_SWITCH); >> + ports_num = (uint8_t) cl_ptr_vector_get_size(&p_group->ports); >> + for (i = 0; i < ports_num; i++) { >> + cl_ptr_vector_at(&p_group->ports, i, (void *)&p_port); >> + if (sw_set_hops(p_remote_sw, cl_ntoh16(target_lid), >> + p_port->remote_port_num, hops)); > > Should ';' be here at end of if()? Oh well, that what happens when doing some cosmetics after testing :( Thanks for spotting it. Wait for V3... -- Yevgeny > Sasha > >> + return -1; >> + } >> + return 0; >> +} >> + >> /*************************************************** >> ** >> ** ftree_hca_t functions > From vlad at lists.openfabrics.org Sat Mar 14 03:21:40 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sat, 14 Mar 2009 03:21:40 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090314-0200 daily build status Message-ID: <20090314102140.EDAF8E60F2D@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From tziporet at mellanox.co.il Sat Mar 14 11:05:48 2009 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Sat, 14 Mar 2009 20:05:48 +0200 Subject: [ofa-general] EWG (OFED) meeting minutes for Mar 12 09 Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD0208103E@mtlexch01.mtl.com> These are the OFED (EWG) meeting minutes for Mar 12 on OFED 1.4 and 1.5 schedule Meeting Summary: ============== 1. OFED 1.4.1: RC1 done; Many bugs to fix; Schedule on track 2. OFED 1.5 will be based over kernel 2.6.30. Release planed for Q3 (end of Aug or Sep) Details: ====== > a. OFED 1.4.1 release status: > Reminder for OFED 1.4.1 schedule: > * RC1 - Mar 4 - done > * RC2 - Mar 17 > * RC3 - Mar 31 > * GA - Apr 7 > > Critical bugs: > bug_id bug_severity op_sys assigned_to short_short_desc > > 1538 blocker Other bugzilla at openib.org how to deal with > /usr/src/ofa_kernel/include/linux/autoconf.h? - Steve will look > 1537 blocker RHEL 5 jsquyres at cisco.com openmpi_gcc compilation > failure - there is a patch - Jeff sent a patch > 1525 blocker RHEL 5 vlad at mellanox.co.il OFED 1.4.1 fails to > compile of EL5.3 on IA64 - Woody to sent a patch > 1546 blocker Other vlad at mellanox.co.il Compilation on SLES11 > (x86_64 archi') failed - Jeff Baker > 1540 critical SLES 10 andy.grover at oracle.com openib stop > hangs on sles10sp2, ppc64 after kernel Oops in rds - maybe fixed? > 1524 critical RHEL 5 eli at mellanox.co.il Running out of > MTT segments, Intel server - 24 cores/64GB memory - fixed > 1547 critical Other eli at mellanox.co.il modules are not > being loaded after openibd restart on SLES11 > 1548 critical Other vlad at mellanox.co.il Can't compile > OFED1.4.1 on SLES11 - PPC - Alexander (IBM) > 1348 major RHEL 5 amirv at mellanox.co.il Sdp sockets doesnt > closed after programs end > 1472 major RHEL 5 amirv at mellanox.co.il kernel panic when trying > to unload sdp or reboot > 1545 major Other eli at mellanox.co.il Performance degradation > in ofed 1.4.1 in TCP BW for some packets size > 1287 major RHEL 5 jackm at mellanox.co.il IPoIB datagram mode > initial packet loss > 1506 major RHEL 5 jackm at mellanox.co.il driver occasionally > fails to start on driver restart > 1539 major All orenk at dev.mellanox.co.il ibutils fails to > compile on ppc64 > 1528 major RHEL 5 jackm at mellanox.co.il IPoIB get stack when > running Hadoop application. - on work > 1529 major RHEL 5 jackm at mellanox.co.il Opensm cannot be stopped > following openib failure. - on work > 1544 major SLES 10 yosefe at voltaire.com mpi_latency in ompenmpi > hangs - nofix > > Betsy raised a concern on the number of bugs. Many bugs are related to the new OSes - RHEL 5.3 and SLES 11. The other bugs are bugs that were found by more QA tests that are done with OFED 1.4 Steve updated that NFS/RDMA backports are ready for RHEL 5.2 and 5.3, and SLES 10 SP2 and SLES 11 All new patches will be taken by Vlad on Sunday. > b. OFED 1.5: > * OFED 1.5 will be based over kernel 2.6.30 (meaning start builds > when 2.6.30-rc1 is available in April) > * OFED 1.5 schedule will be late Q3 - Aug/Sep > Final schedule and dates will be set after Sonoma, based on the inputs we will get. > c. Bill presented some ideas for changes on OFA activities and role. Since the time was short we decided to continue this topic F2F in Sonoma > Tziporet > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kliteyn at dev.mellanox.co.il Sun Mar 15 02:22:17 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Sun, 15 Mar 2009 11:22:17 +0200 Subject: [ofa-general] [PATCH v3] opensm/osm_ucast_ftree.c: some refactoring Message-ID: <49BCC8C9.1020400@dev.mellanox.co.il> Hi Sasha, Some code refactoring: setting entry in the lid matrices should be done per port group. Signed-off-by: Yevgeny Kliteynik --- opensm/opensm/osm_ucast_ftree.c | 66 +++++++++++++++++--------------------- 1 files changed, 30 insertions(+), 36 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index e841368..12be465 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -698,6 +698,27 @@ static inline cl_status_t sw_set_hops(IN ftree_sw_t * p_sw, IN uint16_t lid_ho, return osm_switch_set_hops(p_sw->p_osm_sw, lid_ho, port_num, hops); } +/***************************************************/ + +static int set_hops_on_remote_sw(IN ftree_port_group_t * p_group, + IN ib_net16_t target_lid, + IN uint8_t hops) +{ + ftree_port_t * p_port; + uint8_t i, ports_num; + ftree_sw_t * p_remote_sw = p_group->remote_hca_or_sw.p_sw; + + CL_ASSERT(p_group->remote_node_type == IB_NODE_TYPE_SWITCH); + ports_num = (uint8_t) cl_ptr_vector_get_size(&p_group->ports); + for (i = 0; i < ports_num; i++) { + cl_ptr_vector_at(&p_group->ports, i, (void *)&p_port); + if (sw_set_hops(p_remote_sw, cl_ntoh16(target_lid), + p_port->remote_port_num, hops)) + return -1; + } + return 0; +} + /*************************************************** ** ** ftree_hca_t functions @@ -1988,20 +2009,10 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, /* On the remote switch that is pointed by the p_group, set hops for ALL the ports in the remote group. */ - for (j = 0; j < ports_num; j++) { - cl_ptr_vector_at(&p_group->ports, j, - (void *)&p_port); - - sw_set_hops(p_remote_sw, - cl_ntoh16(target_lid), - p_port->remote_port_num, - ((target_rank - - highest_rank_in_route) + - (p_remote_sw->rank - - highest_rank_in_route) - + reverse_hops * 2)); - } - + set_hops_on_remote_sw(p_group, target_lid, + ((target_rank - highest_rank_in_route) + + (p_remote_sw->rank - highest_rank_in_route) + + reverse_hops * 2)); } /* The number of upgoing routes is tracked in the @@ -2213,18 +2224,9 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* On the remote switch that is pointed by the min_group, set hops for ALL the ports in the remote group. */ - ports_num = - (uint16_t) cl_ptr_vector_get_size(&p_min_group-> - ports); - for (j = 0; j < ports_num; j++) { - cl_ptr_vector_at(&p_min_group->ports, j, - (void *)&p_port); - sw_set_hops(p_remote_sw, - cl_ntoh16(target_lid), - p_port->remote_port_num, - target_rank - p_remote_sw->rank + - 2 * reverse_hops); - } + set_hops_on_remote_sw(p_min_group, target_lid, + target_rank - p_remote_sw->rank + + 2 * reverse_hops); } /* Recursion step: @@ -2305,16 +2307,8 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, /* On the remote switch that is pointed by the p_group, set hops for ALL the ports in the remote group. */ - ports_num = (uint16_t) cl_ptr_vector_get_size(&p_group->ports); - for (j = 0; j < ports_num; j++) { - cl_ptr_vector_at(&p_group->ports, j, (void *)&p_port); - - sw_set_hops(p_remote_sw, - cl_ntoh16(target_lid), - p_port->remote_port_num, - target_rank - p_remote_sw->rank + - 2 * reverse_hops); - } + set_hops_on_remote_sw(p_group, target_lid, + target_rank - p_remote_sw->rank + 2 * reverse_hops); /* Recursion step: Assign downgoing ports by stepping up, starting on REMOTE switch. */ -- 1.5.1.4 From vlad at lists.openfabrics.org Sun Mar 15 03:23:27 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sun, 15 Mar 2009 03:23:27 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090315-0200 daily build status Message-ID: <20090315102328.17722E60FAD@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From purdy at sgi.com Sun Mar 15 07:42:07 2009 From: purdy at sgi.com (Dale Purdy) Date: Sun, 15 Mar 2009 09:42:07 -0500 Subject: [ofa-general] [PATCH] Add LMC support to DOR routing Message-ID: <20090315144207.GA18027@sgi.com> DOR routing equilizes traffic across redundant links representing dimensions. This change makes DOR compatible with LMC and tries to equilize traffic across redundant links between LMC aliased LIDs. Signed-off-by: Dale Purdy --- opensm/include/opensm/osm_switch.h | 4 +++ opensm/opensm/osm_switch.c | 42 +++++++++++++++++++++++------------ opensm/opensm/osm_ucast_mgr.c | 8 +++++- 3 files changed, 37 insertions(+), 17 deletions(-) diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h index 0b15e63..7ce28c5 100644 --- a/opensm/include/opensm/osm_switch.h +++ b/opensm/include/opensm/osm_switch.h @@ -167,6 +167,7 @@ struct osm_remote_guids_count { struct osm_remote_node { osm_node_t *node; unsigned forwarded_to; + uint8_t port; } guids[0]; }; /* @@ -179,6 +180,9 @@ struct osm_remote_guids_count { * * forwarded_to * A count of lids forwarded to this node. +* +* port +* Port number on the node. *********/ /****f* OpenSM: Switch/osm_switch_delete diff --git a/opensm/opensm/osm_switch.c b/opensm/opensm/osm_switch.c index 6dde47c..d236329 100644 --- a/opensm/opensm/osm_switch.c +++ b/opensm/opensm/osm_switch.c @@ -303,6 +303,7 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, osm_node_t *p_rem_node; osm_node_t *p_rem_node_first = NULL; struct osm_remote_node *p_remote_guid = NULL; + struct osm_remote_node null_remote_node = {NULL, 0, 0}; CL_ASSERT(lid_ho > 0); @@ -413,12 +414,37 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, check_count = osm_port_prof_path_count_get(&p_sw->p_prof[port_num]); + if (dor) { + /* Get the Remote Node */ + p_rem_physp = osm_physp_get_remote(p_physp); + p_rem_node = osm_physp_get_node_ptr(p_rem_physp); + /* use the first dimension, but spread traffic + * out among the group of ports representing + * that dimension */ + if (!p_rem_node_first) + p_rem_node_first = p_rem_node; + else if (p_rem_node != p_rem_node_first) + continue; + if (routing_for_lmc) { + struct osm_remote_guids_count *r = p_port->priv; + uint8_t rem_port = osm_physp_get_port_num(p_rem_physp); + int j; + + for (j = 0; j < r->count; j++) { + p_remote_guid = &r->guids[j]; + if ((p_remote_guid->node == p_rem_node) + && (p_remote_guid->port == rem_port)) + break; + } + if (j == r->count) + p_remote_guid = &null_remote_node; + } /* Advanced LMC routing requires tracking of the best port by the node connected to the other side of it. */ - if (routing_for_lmc) { + } else if (routing_for_lmc) { /* Is the sys guid already used ? */ p_remote_guid = osm_switch_find_sys_guid_count(p_sw, p_port->priv, @@ -454,20 +480,6 @@ osm_switch_recommend_path(IN const osm_switch_t * const p_sw, the count is min but also lower then the max subscribed */ if (check_count < least_paths) { - if (dor) { - /* Get the Remote Node */ - p_rem_physp = osm_physp_get_remote(p_physp); - p_rem_node = - osm_physp_get_node_ptr(p_rem_physp); - /* use the first dimension, but spread - * traffic out among the group of ports - * representing that dimension */ - if (port_found) { - if (p_rem_node != p_rem_node_first) - continue; - } else - p_rem_node_first = p_rem_node; - } port_found = TRUE; best_port = port_num; least_paths = check_count; diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c index fe0a446..90d9732 100644 --- a/opensm/opensm/osm_ucast_mgr.c +++ b/opensm/opensm/osm_ucast_mgr.c @@ -186,19 +186,22 @@ __osm_ucast_mgr_process_neighbor(IN osm_ucast_mgr_t * const p_mgr, /********************************************************************** **********************************************************************/ static struct osm_remote_node * -find_and_add_remote_sys(osm_switch_t *sw, uint8_t port, +find_and_add_remote_sys(osm_switch_t *sw, uint8_t port, const boolean_t dor, struct osm_remote_guids_count *r) { unsigned i; osm_physp_t *p = osm_node_get_physp_ptr(sw->p_node, port); osm_node_t *node = p->p_remote_physp->p_node; + uint8_t rem_port = osm_physp_get_port_num(p->p_remote_physp); for (i = 0; i < r->count; i++) if (r->guids[i].node == node) + if (!dor || (r->guids[i].port == rem_port)) return &r->guids[i]; r->guids[i].node = node; r->guids[i].forwarded_to = 0; + r->guids[i].port = rem_port; r->count++; return &r->guids[i]; } @@ -236,7 +239,7 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, if (lid_ho > max_lid_ho) goto Exit; - if (lid_offset) + if (lid_offset && !p_mgr->is_dor) /* ignore potential overflow - it is handled in osm_switch.c */ start_from = osm_switch_get_port_by_lid(p_sw, lid_ho - 1) + 1; @@ -302,6 +305,7 @@ __osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, osm_switch_count_path(p_sw, port); if (port > 0 && p_port->priv && (rem_node_used = find_and_add_remote_sys(p_sw, port, + p_mgr->is_dor, p_port->priv))) rem_node_used->forwarded_to++; } -- 1.6.2 From orenk at dev.mellanox.co.il Sun Mar 15 11:26:50 2009 From: orenk at dev.mellanox.co.il (Oren Kladnitsky) Date: Sun, 15 Mar 2009 20:26:50 +0200 Subject: [ofa-general] Re: [PATCH] Fix credit loop checking In-Reply-To: <20090212151436.GA17309@sgi.com> References: <20090212151436.GA17309@sgi.com> Message-ID: <49BD486A.6010208@dev.mellanox.co.il> Dale Purdy wrote: > ibdiagnet/ibdiagui and ibdmchk assume that up/down routing is being > used if it is able to find roots, whether the root are correct or not. > If it finds roots it does its credit loop validation based on whether > the up/down rules are followed instead of doing a rigorous credit loop > check. If the roots aren't correct, this can lead to determination of > credit loop problems on topologies that don't have problems. ibdmchk > has an option to supply one's own root_guids file to override this if > you actually are using up/down and have your own roots that were used > to set up the routing, but ibdiagnet/ibdiagui doesn't. In any case > there shouldn't be assumptions about what the routing algorithm is, or > what the roots are when checking for credit loops. > > Add a -u option to ibdiagnet/ibdiagui. Change ibdiagnet/ibdiagui and > ibdmchk to only do the simple up/down rule checking against roots when > the -u option is used. Otherwise the full credit loop check is done. > > Signed-off-by: Dale Purdy > --- > ibdiag/doc/ibdiagnet.pod | 11 ++++++++++- > ibdiag/doc/ibdiagui.pod | 2 +- > ibdiag/src/ibdebug.tcl | 11 +++++++---- > ibdiag/src/ibdebug_if.tcl | 10 +++++++--- > ibdm/src/osm_check.cpp | 22 +++++++++------------- > 5 files changed, 34 insertions(+), 22 deletions(-) > > diff --git a/ibdiag/doc/ibdiagnet.pod b/ibdiag/doc/ibdiagnet.pod > index d2cf460..cdc78ed 100644 > --- a/ibdiag/doc/ibdiagnet.pod > +++ b/ibdiag/doc/ibdiagnet.pod > @@ -4,7 +4,7 @@ B > > =head1 SYNOPSYS > > -ibdiagnet [-c ] [-v] [-r] [-o ] > +ibdiagnet [-c ] [-v] [-r] [-u] [-o ] > [-t ] [-s ] [-i ] [-p ] [-wt] > [-pm] [-pc] [-P <=>] > [-lw <1x|4x|12x>] [-ls <2.5|5|10>] > @@ -135,6 +135,15 @@ Provides a report of the fabric qualities > > =back > > +=item B<-u> : > + > +=over > + > +=item > +Credit loop check based on UpDown rules > + > +=back > + > =item B<-t > : > > =over > diff --git a/ibdiag/doc/ibdiagui.pod b/ibdiag/doc/ibdiagui.pod > index 4e0250f..86a2df9 100644 > --- a/ibdiag/doc/ibdiagui.pod > +++ b/ibdiag/doc/ibdiagui.pod > @@ -4,7 +4,7 @@ B > > =head1 SYNOPSYS > > -ibdiagui [-c ] [-v] [-r] [-o ] > +ibdiagui [-c ] [-v] [-r] [-u] [-o ] > [-t ] [-s ] [-i ] [-p ] > [-pm] [-pc] [-P =] > [-lw <1x|4x|12x>] [-ls <2.5|5|10>] > diff --git a/ibdiag/src/ibdebug.tcl b/ibdiag/src/ibdebug.tcl > index 3a464f2..04a8566 100644 > --- a/ibdiag/src/ibdebug.tcl > +++ b/ibdiag/src/ibdebug.tcl > @@ -4391,10 +4391,13 @@ proc DumpFabQualities {} { > inform "-I-ibdiagnet:check.credit.loops.header" > > # report credit loops > - ibdmCalcMinHopTables $fabric > - set roots [ibdmFindRootNodesByMinHop $fabric] > - # just flush out any logs > - set report [ibdmGetAndClearInternalLog] > + set roots "" > + if { [info exists G(argv:updown)] } { > + ibdmCalcMinHopTables $fabric > + set roots [ibdmFindRootNodesByMinHop $fabric] > + # just flush out any logs > + set report [ibdmGetAndClearInternalLog] > + } > if {[llength $roots]} { > inform "-I-reporting:found.roots" $roots > ibdmReportNonUpDownCa2CaPaths $fabric $roots > diff --git a/ibdiag/src/ibdebug_if.tcl b/ibdiag/src/ibdebug_if.tcl > index 21afc45..cf1b571 100644 > --- a/ibdiag/src/ibdebug_if.tcl > +++ b/ibdiag/src/ibdebug_if.tcl > @@ -163,6 +163,10 @@ proc SetInfoArgv {} { > -t,param "topo-file" > -t,desc "Specifies the topology file name" > > + -u,name "updown" > + -u,desc "Indicates that UpDown rule checking should be done against automaticly determined roots" > + -u,arglen 0 > + > -v,name "verbose" > -v,desc "Instructs the tool to run in verbose mode" > -v,arglen 0 > @@ -322,8 +326,8 @@ proc SetToolsFlags {} { > array set TOOLS_FLAGS { > ibping "(n|l|d) . c w v o . t s i p " > ibdiagpath "(n|l|d) . c v o smp . t s i p . pm pc P . lw ls sl ." > - ibdiagui " c v r o . t s i p . pm pc P . lw ls ." > - ibdiagnet " c v r o . t s i p wt . pm pc P . lw ls . skip load_db csv" > + ibdiagui " c v r u o . t s i p . pm pc P . lw ls ." > + ibdiagnet " c v r u o . t s i p wt . pm pc P . lw ls . skip load_db csv" > ibcfg "(n|l|d) (c|q) . t s i p o" > ibmad "(m) (a) (n|l|d) . t s i p o ; (q) a" > ibsac "(m) (a) k . t s i p o ; (q) a" > @@ -2535,7 +2539,7 @@ proc showHelpPage { args } { > Hop-count information: > maximal hop-count, an example path, and a hop-count histogram > All CA-to-CA paths traced > - Credit loop report > + Credit loop report (based on UpDown if -u option is provided) > mgid-mlid-HCAs matching table > Note: In case the IB fabric includes only one CA, then CA-to-CA paths are not > reported. > diff --git a/ibdm/src/osm_check.cpp b/ibdm/src/osm_check.cpp > index 1c18c1c..09a3637 100644 > --- a/ibdm/src/osm_check.cpp > +++ b/ibdm/src/osm_check.cpp > @@ -552,21 +552,17 @@ int main (int argc, char **argv) { > list rootNodes; > int anyErr = 0; > > - if (RootsFileName.size()) > - { > - if (TopoFile.size()) > - { > - rootNodes = ParseRootNodeNamesFile(&fabric, RootsFileName); > - } > - else > - { > - rootNodes = ParseRootNodeGuidsFile(&fabric, RootsFileName); > - } > - } > - else > - { > + if (UseUpDown) { > + if (RootsFileName.size()) { > + if (TopoFile.size()) { > + rootNodes = ParseRootNodeNamesFile(&fabric, RootsFileName); > + } else { > + rootNodes = ParseRootNodeGuidsFile(&fabric, RootsFileName); > + } > + } else { > rootNodes = SubnMgtFindRootNodesByMinHop(&fabric); > } > + } > > if (!rootNodes.empty()) { > cout << "-I- Recognized " << rootNodes.size() << " root nodes:" << endl; > Hi Dale, The above patch broke osm_check.cpp . Attached is the patch I used for this purpose. Thanks, Oren. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 0001-Provide-a-new-flag-u-updn-to-ibdiagnet-and-ibdmch.patch URL: From Jeffrey.C.Becker at nasa.gov Sun Mar 15 11:30:41 2009 From: Jeffrey.C.Becker at nasa.gov (Jeff Becker) Date: Sun, 15 Mar 2009 11:30:41 -0700 Subject: [ofa-general] EWG (OFED) meeting minutes for Mar 12 09 In-Reply-To: <5D49E7A8952DC44FB38C38FA0D758EAD0208103E@mtlexch01.mtl.com> References: <5D49E7A8952DC44FB38C38FA0D758EAD0208103E@mtlexch01.mtl.com> Message-ID: <49BD4951.3000500@nasa.gov> Tziporet Koren wrote: > > These are the OFED (EWG) meeting minutes for Mar 12 on OFED 1.4 and > 1.5 schedule > > Meeting Summary: > > ============== > > 1. OFED 1.4.1: RC1 done; Many bugs to fix; Schedule on track > > 2. OFED 1.5 will be based over kernel 2.6.30. Release planed for Q3 > (end of Aug or Sep) > > Details: > > ====== > > a. OFED 1.4.1 release status: > > Reminder for OFED 1.4.1 schedule: > > o RC1 - Mar 4 - done > o RC2 - Mar 17 > o RC3 - Mar 31 > o GA - Apr 7 > --snip-- > Steve updated that NFS/RDMA backports are ready for RHEL 5.2 and 5.3, > and SLES 10 SP2 and SLES 11 > Small correction: I'm still integrating Jon Mason's RHEL5.2 backport of NFSRDMA into SLES10SP2. Thanks. -jeff > All new patches will be taken by Vlad on Sunday. > > b. OFED 1.5: > > o OFED 1.5 will be based over kernel 2.6.30 (meaning start > builds when 2.6.30-rc1 is available in April) > o OFED 1.5 schedule will be late Q3 - Aug/Sep > > Final schedule and dates will be set after Sonoma, based on the inputs > we will get. > > c. Bill presented some ideas for changes on OFA activities and role. > > Since the time was short we decided to continue this topic F2F in Sonoma > > > Tziporet > > From bart.vanassche at gmail.com Sun Mar 15 11:55:49 2009 From: bart.vanassche at gmail.com (Bart Van Assche) Date: Sun, 15 Mar 2009 19:55:49 +0100 Subject: ***SPAM*** Re: [ofa-general] EWG (OFED) meeting minutes for Mar 12 09 In-Reply-To: <5D49E7A8952DC44FB38C38FA0D758EAD0208103E@mtlexch01.mtl.com> References: <5D49E7A8952DC44FB38C38FA0D758EAD0208103E@mtlexch01.mtl.com> Message-ID: On Sat, Mar 14, 2009 at 7:05 PM, Tziporet Koren wrote: > 1348    major   RHEL 5  amirv at mellanox.co.il    Sdp sockets doesnt closed after programs end As far as I know SDP is currently implemented in OFED as a separate address family (AF_INET_SDP). This is an unfortunate approach because: * This approach will never be accepted upstream by the Linux kernel maintainers. * The approach of of preloading a library in order to make applications use SDP without modifying these applications is error prone -- it is really hard to make such a library 100% correct. When will work start on an SDP API that is acceptable for inclusion in the mainstream Linux kernel ? One possible approach is to extend the BSD socket API with support for multiple IP stack implementations. This can be implemented by e.g. adding a system call msocket() that has four parameters -- the three classic socket() parameters and a fourth parameter for the IP stack. See also http://wiki.virtualsquare.org/index.php/Multi_stack_support_for_Berkeley_Sockets or http://www.fosdem.org/2009/schedule/events/ipn_msockets for a more complete explanation. A previous discussion about SDP can be found here: http://www.mail-archive.com/netdev at vger.kernel.org/msg08546.html Bart. From rdreier at cisco.com Sun Mar 15 13:42:43 2009 From: rdreier at cisco.com (Roland Dreier) Date: Sun, 15 Mar 2009 13:42:43 -0700 Subject: ***SPAM*** Re: [ofa-general] EWG (OFED) meeting minutes for Mar 12 09 In-Reply-To: (Bart Van Assche's message of "Sun, 15 Mar 2009 19:55:49 +0100") References: <5D49E7A8952DC44FB38C38FA0D758EAD0208103E@mtlexch01.mtl.com> Message-ID: > As far as I know SDP is currently implemented in OFED as a separate > address family (AF_INET_SDP). This is an unfortunate approach because: > * This approach will never be accepted upstream by the Linux kernel maintainers. > One possible approach is to extend the BSD socket API with support for > multiple IP stack implementations. This can be implemented by e.g. > adding a system call msocket() that has four parameters -- the three > classic socket() parameters and a fourth parameter for the IP stack. I'm not sure why you make this assertion... a new protocol family for SOCK_STREAM sockets seems far more likely to be accepted upstream than adding support for multiple networking stacks. For example RDS was recently queued for 2.6.30 as a new protocol family. I have a hard time imagining multiple IP stacks being accepted upstream on the other hand. - R. From dorfman.eli at gmail.com Mon Mar 16 01:24:36 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Mon, 16 Mar 2009 10:24:36 +0200 Subject: [ofa-general] Re: [PATCH v2] opensm: support routing engine update In-Reply-To: <20090312160528.GW8818@sashak.voltaire.com> References: <49A6B618.1090300@gmail.com> <49A6B6EB.80700@gmail.com> <20090312132137.GB8818@sashak.voltaire.com> <49B92C27.7060904@gmail.com> <20090312160528.GW8818@sashak.voltaire.com> Message-ID: <49BE0CC4.6030600@gmail.com> support routing engine update perform setup of routing engine only before configuration. save last routing engine before routing engine selection and cleanup of last if not used any more. Signed-off-by: Eli Dorfman --- opensm/include/opensm/osm_opensm.h | 5 +++ opensm/opensm/osm_opensm.c | 58 +++++++++++++++++++++++++++++++----- opensm/opensm/osm_subnet.c | 7 ++++- opensm/opensm/osm_ucast_mgr.c | 29 ++++++++++++++++++ 4 files changed, 90 insertions(+), 9 deletions(-) diff --git a/opensm/include/opensm/osm_opensm.h b/opensm/include/opensm/osm_opensm.h index c121be4..d297dd8 100644 --- a/opensm/include/opensm/osm_opensm.h +++ b/opensm/include/opensm/osm_opensm.h @@ -122,6 +122,8 @@ typedef enum _osm_routing_engine_type { struct osm_routing_engine { const char *name; void *context; + int initialized; + int (*setup) (void *re, void *p_osm); int (*build_lid_matrices) (void *context); int (*ucast_build_fwd_tables) (void *context); void (*ucast_dump_tables) (void *context); @@ -183,6 +185,7 @@ typedef struct osm_opensm { cl_dispatcher_t disp; cl_plock_t lock; struct osm_routing_engine *routing_engine_list; + struct osm_routing_engine *last_routing_engine; osm_routing_engine_type_t routing_engine_used; osm_stats_t stats; osm_console_t console; @@ -523,5 +526,7 @@ extern volatile unsigned int osm_exit_flag; * Set to one to cause all threads to leave *********/ +void osm_update_routing_engines(osm_opensm_t *osm, const char *engine_names); + END_C_DECLS #endif /* _OSM_OPENSM_H_ */ diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c index cfe6474..7126658 100644 --- a/opensm/opensm/osm_opensm.c +++ b/opensm/opensm/osm_opensm.c @@ -169,14 +169,7 @@ static void setup_routing_engine(osm_opensm_t *osm, const char *name) memset(re, 0, sizeof(struct osm_routing_engine)); re->name = m->name; - if (m->setup(re, osm)) { - OSM_LOG(&osm->log, OSM_LOG_VERBOSE, - "setup of routing" - " engine \'%s\' failed\n", name); - return; - } - OSM_LOG(&osm->log, OSM_LOG_DEBUG, - "\'%s\' routing engine set up\n", re->name); + re->setup = (int (*)(void *, void *))m->setup; append_routing_engine(osm, re); return; } @@ -236,6 +229,55 @@ static void destroy_routing_engines(osm_opensm_t *osm) r->delete(r->context); free(r); } + osm->routing_engine_list = NULL; +} + +static void update_routing_engine( + struct osm_routing_engine *cur, + struct osm_routing_engine *last) +{ + cur->context = last->context; + cur->initialized = last->initialized; + cur->setup = last->setup; + cur->build_lid_matrices = last->build_lid_matrices; + cur->ucast_build_fwd_tables = last->ucast_build_fwd_tables; + cur->ucast_dump_tables = last->ucast_dump_tables; + cur->delete = last->delete; +} + +void osm_update_routing_engines(osm_opensm_t *osm, const char *engine_names) +{ + struct osm_routing_engine *r, *l; + + /* find used routing engine and save as last */ + l = r = osm->routing_engine_list; + if (r && osm->routing_engine_used == osm_routing_engine_type(r->name)) { + osm->last_routing_engine = r; + osm->routing_engine_list = r->next; + } + else while ((r = r->next)) { + if (osm->routing_engine_used == + osm_routing_engine_type(r->name)) { + osm->last_routing_engine = r; + l->next = r->next; + break; + } + l = r; + } + /* cleanup prev routing engine list and replace with current list */ + destroy_routing_engines(osm); + setup_routing_engines(osm, engine_names); + + /* check if last routing engine exist in new list and update callbacks */ + r = osm->routing_engine_list; + while (r) { + if (osm->routing_engine_used == + osm_routing_engine_type(r->name)) { + update_routing_engine(r, osm->last_routing_engine); + break; + } + r = r->next; + } } /********************************************************************** diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index 4ab440b..62b7bc7 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -151,6 +151,11 @@ static void opts_setup_sm_priority(osm_subn_t *p_subn, void *p_val) osm_set_sm_priority(p_sm, sm_priority); } +static void opts_setup_routing_engine(osm_subn_t *p_subn, void *p_val) +{ + osm_update_routing_engines(p_subn->p_osm, p_val); +} + static void opts_parse_net64(IN osm_subn_t *p_subn, IN char *p_key, IN char *p_val_str, void *p_v1, void *p_v2, void (*pfn)(osm_subn_t *, void *)) @@ -325,7 +330,7 @@ static const opt_rec_t opt_tbl[] = { { "hop_weights_file", OPT_OFFSET(hop_weights_file), opts_parse_charp, NULL, 0 }, { "port_profile_switch_nodes", OPT_OFFSET(port_profile_switch_nodes), opts_parse_boolean, NULL, 1 }, { "sweep_on_trap", OPT_OFFSET(sweep_on_trap), opts_parse_boolean, NULL, 1 }, - { "routing_engine", OPT_OFFSET(routing_engine_names), opts_parse_charp, NULL, 0 }, + { "routing_engine", OPT_OFFSET(routing_engine_names), opts_parse_charp, opts_setup_routing_engine, 1 }, { "connect_roots", OPT_OFFSET(connect_roots), opts_parse_boolean, NULL, 1 }, { "use_ucast_cache", OPT_OFFSET(use_ucast_cache), opts_parse_boolean, NULL, 1 }, { "log_file", OPT_OFFSET(log_file), opts_parse_charp, NULL, 0 }, diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c index fe0a446..abae978 100644 --- a/opensm/opensm/osm_ucast_mgr.c +++ b/opensm/opensm/osm_ucast_mgr.c @@ -970,8 +970,25 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) p_osm->routing_engine_used = OSM_ROUTING_ENGINE_TYPE_NONE; while (p_routing_eng) { + if (!p_routing_eng->initialized && + p_routing_eng->setup(p_routing_eng, p_osm)) { + OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, + "ERR 3A0F: setup of routing engine \'%s\' failed\n", + p_routing_eng->name); + p_routing_eng = p_routing_eng->next; + continue; + } + OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, + "\'%s\' routing engine set up\n", p_routing_eng->name); + p_routing_eng->initialized = 1; + if (!ucast_mgr_route(p_routing_eng, p_osm)) break; + + /* delete unused routing engine */ + if (p_routing_eng->delete) + p_routing_eng->delete(p_routing_eng->context); + p_routing_eng = p_routing_eng->next; } @@ -982,6 +999,18 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) p_osm->routing_engine_used = OSM_ROUTING_ENGINE_TYPE_MINHOP; } + /* if for some reason different routing engine is used */ + /* cleanup last unused routing engine */ + p_routing_eng = p_osm->last_routing_engine; + if (p_routing_eng) { + if (p_routing_eng->initialized && + p_routing_eng->delete && + p_osm->routing_engine_used != + osm_routing_engine_type(p_routing_eng->name)) + p_routing_eng->delete(p_routing_eng->context); + free(p_routing_eng); + } + OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, "%s tables configured on all switches\n", osm_routing_engine_type_str(p_osm->routing_engine_used)); -- 1.5.5 From dorfman.eli at gmail.com Mon Mar 16 01:27:03 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Mon, 16 Mar 2009 10:27:03 +0200 Subject: ***SPAM*** Re: [ofa-general] [PATCH v4] fix local port smlid In-Reply-To: <20090312161228.GX8818@sashak.voltaire.com> References: <49AFCCC0.6090507@gmail.com> <49AFE1E7.3030803@gmail.com> <49AFE65E.3080809@gmail.com> <49AFE7C0.5060707@gmail.com> <20090312154954.GU8818@sashak.voltaire.com> <49B93119.6000207@gmail.com> <20090312161228.GX8818@sashak.voltaire.com> Message-ID: <49BE0D57.3030509@gmail.com> Sasha Khapyorsky wrote: > On 17:58 Thu 12 Mar , Eli Dorfman (Voltaire) wrote: >>> Instead of touching port_info I would suggest to setup destination lid >>> using smport which you already have: >>> >>> madw->mad_addr.dest_lid = smport->p_physp->port_info.base_lid; >>> >> Local standby SM is polling the master SM, so if for some reason the master >> decides to relinquish mastership then local SM will rediscover the fabric and >> find the new SM. > > Ok. So? Do you want another patch or you'll fix it locally? Eli > >> Also, it is better to update portinfo in case others will attempt to use pi->master_sm_lid. > > Using outdated PortInfo (and when SM is in Stanby it *is* outdated) is > an error, so nobody should use it. > > Sasha From dorfman.eli at gmail.com Mon Mar 16 01:33:25 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Mon, 16 Mar 2009 10:33:25 +0200 Subject: [ofa-general] ***SPAM*** [PATCH TRIVIAL] opensm/osm_opensm.c: add newline to log message Message-ID: <49BE0ED5.3030702@gmail.com> add newline to log message Signed-off-by: Eli Dorfman --- opensm/opensm/osm_opensm.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c index 7126658..732f427 100644 --- a/opensm/opensm/osm_opensm.c +++ b/opensm/opensm/osm_opensm.c @@ -176,7 +176,7 @@ static void setup_routing_engine(osm_opensm_t *osm, const char *name) } OSM_LOG(&osm->log, OSM_LOG_ERROR, - "cannot find or setup routing engine \'%s\'", name); + "cannot find or setup routing engine \'%s\'\n", name); } static void setup_routing_engines(osm_opensm_t *osm, const char *engine_names) -- 1.5.5 From sashak at voltaire.com Mon Mar 16 02:09:20 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Mar 2009 11:09:20 +0200 Subject: [ofa-general] [PATCH v4] fix local port smlid In-Reply-To: <49BE0D57.3030509@gmail.com> References: <49AFCCC0.6090507@gmail.com> <49AFE1E7.3030803@gmail.com> <49AFE65E.3080809@gmail.com> <49AFE7C0.5060707@gmail.com> <20090312154954.GU8818@sashak.voltaire.com> <49B93119.6000207@gmail.com> <20090312161228.GX8818@sashak.voltaire.com> <49BE0D57.3030509@gmail.com> Message-ID: <20090316090904.GA12557@sashak.voltaire.com> On 10:27 Mon 16 Mar , Eli Dorfman (Voltaire) wrote: > > Do you want another patch or you'll fix it locally? I can fix on my side. Sasha From dorfman.eli at gmail.com Mon Mar 16 02:20:38 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Mon, 16 Mar 2009 11:20:38 +0200 Subject: [ofa-general] ***SPAM*** [PATCH] Added send trap for trap 129 (local link integrity) Message-ID: <49BE19E6.6060101@gmail.com> Added send trap for trap 129 (local link integrity). Signed-off-by: Julia Volynsky --- infiniband-diags/src/ibsendtrap.c | 45 +++++++++++++++++++++++++++++++++++- 1 files changed, 43 insertions(+), 2 deletions(-) diff --git a/infiniband-diags/src/ibsendtrap.c b/infiniband-diags/src/ibsendtrap.c index f2b3e67..f867a6c 100644 --- a/infiniband-diags/src/ibsendtrap.c +++ b/infiniband-diags/src/ibsendtrap.c @@ -48,6 +48,8 @@ #include "ibdiag_common.h" struct ibmad_port *srcport; +/* for local link integrity */ +int error_port = 1; static int send_144_node_desc_update(void) { @@ -85,13 +87,48 @@ static int send_144_node_desc_update(void) return (mad_send_via(&trap_rpc, &sm_port, NULL, ¬ice, srcport)); } +static int send_129_local_link_integrity(void) +{ + ib_portid_t sm_port; + ib_portid_t selfportid; + int selfport; + ib_rpc_t trap_rpc; + ib_mad_notice_attr_t notice; + + if (ib_resolve_self_via(&selfportid, &selfport, NULL, srcport)) + IBERROR("can't resolve self"); + + if (ib_resolve_smlid_via(&sm_port, 0, srcport)) + IBERROR("can't resolve SM destination port"); + + memset(&trap_rpc, 0, sizeof(trap_rpc)); + trap_rpc.mgtclass = IB_SMI_CLASS; + trap_rpc.method = IB_MAD_METHOD_TRAP; + trap_rpc.trid = mad_trid(); + trap_rpc.attr.id = NOTICE; + trap_rpc.datasz = IB_SMP_DATA_SIZE; + trap_rpc.dataoffs = IB_SMP_DATA_OFFS; + + memset(¬ice, 0, sizeof(notice)); + notice.generic_type = 0x80 | IB_NOTICE_TYPE_INFO; + notice.g_or_v.generic.prod_type_lsb = cl_hton16(IB_NODE_TYPE_CA); + notice.g_or_v.generic.trap_num = cl_hton16(129); + notice.issuer_lid = cl_hton16((uint16_t) selfportid.lid); + notice.data_details.ntc_129_131.lid = cl_hton16((uint16_t) selfportid.lid); + notice.data_details.ntc_129_131.pad = 0; + notice.data_details.ntc_129_131.port_num = error_port; + + return (mad_send_via(&trap_rpc, &sm_port, NULL, ¬ice, srcport)); +} + typedef struct _trap_def { char *trap_name; int (*send_func) (void); } trap_def_t; -trap_def_t traps[2] = { +trap_def_t traps[3] = { {"node_desc_change", send_144_node_desc_update}, + {"local_link_integrity", send_129_local_link_integrity}, {NULL, NULL} }; @@ -115,7 +152,7 @@ int main(int argc, char **argv) char *trap_name = NULL; int i, n, rc; - n = sprintf(usage_args, "[]\n" + n = sprintf(usage_args, "[] []\n" "\nArgument can be one of the following:\n"); for (i = 0; traps[i].trap_name; i++) { n += snprintf(usage_args + n, sizeof(usage_args) - n, @@ -138,6 +175,10 @@ int main(int argc, char **argv) trap_name = argv[0]; } + if (argc > 1) { + error_port = atoi(argv[1]); + } + madrpc_show_errors(1); srcport = mad_rpc_open_port(ibd_ca, ibd_ca_port, mgmt_classes, 2); -- 1.5.5 From sashak at voltaire.com Mon Mar 16 02:19:02 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Mar 2009 11:19:02 +0200 Subject: [ofa-general] [PATCH v4] fix local port smlid In-Reply-To: <49AFE7C0.5060707@gmail.com> References: <49AFCCC0.6090507@gmail.com> <49AFE1E7.3030803@gmail.com> <49AFE65E.3080809@gmail.com> <49AFE7C0.5060707@gmail.com> Message-ID: <20090316091855.GB12557@sashak.voltaire.com> On 16:54 Thu 05 Mar , Eli Dorfman (Voltaire) wrote: > > fix local port smlid > > when opensm goes from master to standby due to handover its local > port smlid will be wrong (equal to its own lid). > the new master opensm will change smlid only after local sm > (previous master) is in standby and will not query its port again. > > Signed-off-by: Eli Dorfman Applied (with discussed changes). Thanks. Sasha From sashak at voltaire.com Mon Mar 16 02:19:55 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Mar 2009 11:19:55 +0200 Subject: [ofa-general] Re: [PATCH v2] opensm: set IS_SM bit during opensm init In-Reply-To: <49B918EE.5010504@gmail.com> References: <49AFC929.20606@gmail.com> <49B918EE.5010504@gmail.com> Message-ID: <20090316091955.GC12557@sashak.voltaire.com> On 16:15 Thu 12 Mar , Eli Dorfman (Voltaire) wrote: > > Set IS_SM bit during opensm init > > Set IS_SM bit during opensm initalization. > This fixes a problem of SM that enters to standby and has > mismatch between local capability (IS_SM off) and the SMA. > > Signed-off-by: Eli Dorfman Applied. Thanks. Sasha From sashak at voltaire.com Mon Mar 16 02:33:08 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Mon, 16 Mar 2009 11:33:08 +0200 Subject: [ofa-general] Re: [PATCH v3] opensm/osm_ucast_ftree.c: some refactoring In-Reply-To: <49BCC8C9.1020400@dev.mellanox.co.il> References: <49BCC8C9.1020400@dev.mellanox.co.il> Message-ID: <20090316093308.GD12557@sashak.voltaire.com> On 11:22 Sun 15 Mar , Yevgeny Kliteynik wrote: > Hi Sasha, > > Some code refactoring: setting entry in the lid > matrices should be done per port group. > > Signed-off-by: Yevgeny Kliteynik Applied. Thanks. Sasha From vlad at lists.openfabrics.org Mon Mar 16 03:28:19 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Mon, 16 Mar 2009 03:28:19 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090316-0200 daily build status Message-ID: <20090316102819.B1E5BE609E2@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From hnrose at comcast.net Mon Mar 16 05:39:35 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Mon, 16 Mar 2009 07:39:35 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] opensm/osm_req.c: Update log message based on commit 3551389dcb7353ffd51c66e6ad518648bc1dd19e Message-ID: <20090316123935.GA26174@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_req.c b/opensm/opensm/osm_req.c index a01750c..6eb8213 100644 --- a/opensm/opensm/osm_req.c +++ b/opensm/opensm/osm_req.c @@ -283,7 +283,8 @@ int osm_send_trap144(osm_sm_t *sm, ib_net16_t local) OSM_LOG(sm->p_log, OSM_LOG_DEBUG, "Sending Trap 144, TID 0x%" PRIx64 " to SM lid %u\n", - cl_ntoh64(smp->trans_id), cl_ntoh16(pi->master_sm_base_lid)); + cl_ntoh64(smp->trans_id), + cl_ntoh16(smport->p_physp->port_info.base_lid)); osm_vl15_post(sm->p_vl15, madw); From bart.vanassche at gmail.com Mon Mar 16 05:02:42 2009 From: bart.vanassche at gmail.com (Bart Van Assche) Date: Mon, 16 Mar 2009 13:02:42 +0100 Subject: ***SPAM*** Re: ***SPAM*** Re: [ofa-general] EWG (OFED) meeting minutes for Mar 12 09 In-Reply-To: References: <5D49E7A8952DC44FB38C38FA0D758EAD0208103E@mtlexch01.mtl.com> Message-ID: On Sun, Mar 15, 2009 at 9:42 PM, Roland Dreier wrote: >  > As far as I know SDP is currently implemented in OFED as a separate >  > address family (AF_INET_SDP). This is an unfortunate approach because: >  > * This approach will never be accepted upstream by the Linux kernel maintainers. > >  > One possible approach is to extend the BSD socket API with support for >  > multiple IP stack implementations. This can be implemented by e.g. >  > adding a system call msocket() that has four parameters -- the three >  > classic socket() parameters and a fourth parameter for the IP stack. > > I'm not sure why you make this assertion... a new protocol family for > SOCK_STREAM sockets seems far more likely to be accepted upstream than > adding support for multiple networking stacks.  For example RDS was > recently queued for 2.6.30 as a new protocol family.  I have a hard time > imagining multiple IP stacks being accepted upstream on the other hand. I would like to clarify that I do not know the opinion of the network subsystem maintainers about msocket(), but that I posted this information only as an example of an alternative for AF_INET_SDP. Bart. From Line.Holen at Sun.COM Mon Mar 16 05:17:20 2009 From: Line.Holen at Sun.COM (Line.Holen at Sun.COM) Date: Mon, 16 Mar 2009 13:17:20 +0100 Subject: [ofa-general] [PATCH] opensm/osm_link_mgr.c Remove __osm_ prefix Message-ID: <49BE4350.4000001@Sun.COM> Remove all __osm_ prefixes from static functions Signed-off-by: Line Holen --- diff --git a/opensm/opensm/osm_link_mgr.c b/opensm/opensm/osm_link_mgr.c index c88da29..d3fb7f1 100644 --- a/opensm/opensm/osm_link_mgr.c +++ b/opensm/opensm/osm_link_mgr.c @@ -94,9 +94,9 @@ link_mgr_get_smsl(IN osm_sm_t * sm, IN osm_physp_t * const p_physp) /********************************************************************** **********************************************************************/ static boolean_t -__osm_link_mgr_set_physp_pi(osm_sm_t * sm, - IN osm_physp_t * const p_physp, - IN uint8_t const port_state) +link_mgr_set_physp_pi(osm_sm_t * sm, + IN osm_physp_t * const p_physp, + IN uint8_t const port_state) { uint8_t payload[IB_SMP_DATA_SIZE]; ib_port_info_t *const p_pi = (ib_port_info_t *) payload; @@ -406,9 +406,9 @@ Exit: /********************************************************************** **********************************************************************/ static osm_signal_t -__osm_link_mgr_process_node(osm_sm_t * sm, - IN osm_node_t * const p_node, - IN const uint8_t link_state) +link_mgr_process_node(osm_sm_t * sm, + IN osm_node_t * const p_node, + IN const uint8_t link_state) { uint32_t i; uint32_t num_physp; @@ -454,7 +454,7 @@ __osm_link_mgr_process_node(osm_sm_t * sm, "Physical port %u already %s. Skipping\n", p_physp->port_num, ib_get_port_state_str(current_state)); - else if (__osm_link_mgr_set_physp_pi(sm, p_physp, link_state)) + else if (link_mgr_set_physp_pi(sm, p_physp, link_state)) signal = OSM_SIGNAL_DONE_PENDING; } @@ -479,7 +479,7 @@ osm_signal_t osm_link_mgr_process(osm_sm_t * sm, IN const uint8_t link_state) for (p_node = (osm_node_t *) cl_qmap_head(p_node_guid_tbl); p_node != (osm_node_t *) cl_qmap_end(p_node_guid_tbl); p_node = (osm_node_t *) cl_qmap_next(&p_node->map_item)) { - if (__osm_link_mgr_process_node(sm, p_node, link_state) == + if (link_mgr_process_node(sm, p_node, link_state) == OSM_SIGNAL_DONE_PENDING) signal = OSM_SIGNAL_DONE_PENDING; } From michael.heinz at qlogic.com Mon Mar 16 07:06:27 2009 From: michael.heinz at qlogic.com (Mike Heinz) Date: Mon, 16 Mar 2009 09:06:27 -0500 Subject: [ofa-general] OPENIB/OFED Tutorial? Message-ID: <4C2744E8AD2982428C5BFE523DF8CDCB43E6D5A4DC@MNEXMB1.qlogic.org> Has anyone ever written a tutorial that I can give new coders to show them the basic steps of setting up a simple RC connection in OFED? -- Michael Heinz Principal Engineer, Qlogic Corporation King of Prussia, Pennsylvania -------------- next part -------------- An HTML attachment was scrubbed... URL: From sumeetgandhare at gmail.com Mon Mar 16 08:38:25 2009 From: sumeetgandhare at gmail.com (sumeet gandhare) Date: Mon, 16 Mar 2009 21:08:25 +0530 Subject: [ofa-general] entry points in OFED kernel module Message-ID: Hi, I am new to OFED stack and wish to understand the design from the source code. I have read the chapter 3 of IBA spec from OpenFabrics and understand the concept of Queue Pair, Completion Queue, work request, consumer, LID, GUID. Basically in IB, the consumer process enqueues the work request to the send queue pair which specifies the virtual address of the message to be sent and destination LID and QPN. The receiving consumer enqueues work elements to receive queue specifying the virtual address of the message being received. On arrival of the message , the completion queue contains a CQE notification to the client. So hardware basically is a DMA engine which transfer the message from source virtual address to desitnation virtual address. I wish to understand how this part better. Please could you suggest entry points in the stack like verbs and the interaction with device driver like iPath and Mlx.. What all files I should look at and in what sequence to better understand the programming model and its implementation to send and receive the IB packets on the wire? Thanks Sumeet -------------- next part -------------- An HTML attachment was scrubbed... URL: From hnrose at comcast.net Mon Mar 16 09:52:25 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Mon, 16 Mar 2009 11:52:25 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] libibmad/libibmad.map: Eliminate perf_classportinfo_query_via Message-ID: <20090316165225.GA7793@comcast.net> since it was eliminated by commit 6961ab9e34332f88a2688866b37e6268f9d587f6 Signed-off-by: Hal Rosenstock --- diff --git a/libibmad/src/libibmad.map b/libibmad/src/libibmad.map index b7c2031..4306dbc 100644 --- a/libibmad/src/libibmad.map +++ b/libibmad/src/libibmad.map @@ -102,6 +102,5 @@ IBMAD_1.3 { ib_resolve_guid_via; ib_resolve_portid_str_via; ib_resolve_self_via; - perf_classportinfo_query_via; local: *; }; From diego.guella at sircomtech.com Mon Mar 16 08:59:13 2009 From: diego.guella at sircomtech.com (Diego Guella) Date: Mon, 16 Mar 2009 16:59:13 +0100 Subject: [ofa-general] Concurrent RDMA Write w/imm between hosts and bandwidth/latency Message-ID: <005d01c9a650$26bbdbd0$0100000a@DIEGO> Hi all, I am facing this problem: I have 3 hosts A, B, C. Hosts A and B send each other some small buffers at very regular intervals of time, and these buffers are "urgent" or "time critical" buffers. Host C sends to host B a very big buffer, and this is a "non urgent" buffer. I want the "urgent" transfer between A and B to not be affected by this "non urgent" transfer between C and B. The "urgent" buffers should have the same travel times as if the "non urgent" transfer is non-existent. How can I achieve this? Fence indicator? Any hint is greatly appreciated. All the communications are done with RDMA Write with Immediate. The cards I have are MHES14-XTC and MHGA28-XTC. I am using Windows XP 32bit (I am including general list because I think this is a non-windows only "problem") Best regards, Diego -------------- next part -------------- An HTML attachment was scrubbed... URL: From michael.heinz at qlogic.com Mon Mar 16 09:19:02 2009 From: michael.heinz at qlogic.com (Mike Heinz) Date: Mon, 16 Mar 2009 11:19:02 -0500 Subject: [ofa-general] RE: OPENIB/OFED Tutorial? In-Reply-To: <4C2744E8AD2982428C5BFE523DF8CDCB43E6D5A4DC@MNEXMB1.qlogic.org> References: <4C2744E8AD2982428C5BFE523DF8CDCB43E6D5A4DC@MNEXMB1.qlogic.org> Message-ID: <4C2744E8AD2982428C5BFE523DF8CDCB43E6D5A502@MNEXMB1.qlogic.org> To be clear, since this is meant as a teaching tool I'm looking for some of the more low-level details like how to fill out the qp attributes, rather than just using the rdma_cm. -- Michael Heinz Principal Engineer, Qlogic Corporation King of Prussia, Pennsylvania From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Mike Heinz Sent: Monday, March 16, 2009 10:06 AM To: general at lists.openfabrics.org Subject: [ofa-general] OPENIB/OFED Tutorial? Has anyone ever written a tutorial that I can give new coders to show them the basic steps of setting up a simple RC connection in OFED? -- Michael Heinz Principal Engineer, Qlogic Corporation King of Prussia, Pennsylvania -------------- next part -------------- An HTML attachment was scrubbed... URL: From dotanba at gmail.com Mon Mar 16 09:31:37 2009 From: dotanba at gmail.com (Dotan Barak) Date: Mon, 16 Mar 2009 18:31:37 +0200 Subject: ***SPAM*** Re: [ofa-general] Concurrent RDMA Write w/imm between hosts and bandwidth/latency In-Reply-To: <005d01c9a650$26bbdbd0$0100000a@DIEGO> References: <005d01c9a650$26bbdbd0$0100000a@DIEGO> Message-ID: <2f3bf9a60903160931l3fc8b736j3b0d7ddc4fe96169@mail.gmail.com> Hi. Why won't you connect the QPs to different CQs? This way, you will be able to pull the "urgent" CQ first, and only if it is empty, poll the "non-urgent" CQ. Dotan 2009/3/16 Diego Guella : > Hi all, > > I am facing this problem: I have 3 hosts A, B, C. > Hosts A and B send each other some small buffers at very regular intervals > of time, and these buffers are "urgent" or "time critical" buffers. > Host C sends to host B a very big buffer, and this is a "non urgent" buffer. > I want the "urgent" transfer between A and B to not be affected by this "non > urgent" transfer between C and B. > The "urgent" buffers should have the same travel times as if the "non > urgent" transfer is non-existent. > > How can I achieve this? Fence indicator? > Any hint is greatly appreciated. > > > All the communications are done with RDMA Write with Immediate. > The cards I have are MHES14-XTC and MHGA28-XTC. > I am using Windows XP 32bit (I am including general list because I think > this is a non-windows only "problem") > > Best regards, > Diego > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From arlin.r.davis at intel.com Mon Mar 16 16:41:56 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Mon, 16 Mar 2009 16:41:56 -0700 Subject: [ofa-general] [ANNOUNCE] compat-dapl-1.2.13 and dapl-2.0.16 Release Message-ID: New DAPL releases now available from OFA download page: http://www.openfabrics.org/downloads/dapl/ Latest Packages: md5sum: 74974ec74e3d4c254698af240ba5e415 compat-dapl-1.2.13.tar.gz md5sum: 890e22b42397fac9c0c43a97507d1d4c dapl-2.0.16.tar.gz Vlad, please pick up new packages and install following for OFED 1.4.1 RC2: For 1.2 and 2.0 support on same system, including development, install RPM packages as follow: compat-dapl-1.2.13-1 compat-dapl-devel-1.2.13-1 dapl-2.0.16-1 dapl-utils-2.0.16-1 dapl-devel-2.0.16-1 dapl-debuginfo-2.0.16-1 Summary of changes for lastest release: v1,v2 - spec files missing Requires(post) statements for sed/coreutils v1,v2 - dtest/dapltest: use $(top_builddir) for .la files during test builds v1,v2 - scm: remove unecessary thread when using direct objects v2 - scm: remove query gid/lid from connection setup phase v2 - common: add optional 64 bit counters for IA, EP, EVD's (with -DDAPL_COUNTERS) v2 - dtestx: add option -p to print and query new counters v2 - WinOF and OFED: sync up trees for common code base. -arlin From arkady.kanevsky at gmail.com Mon Mar 16 17:09:21 2009 From: arkady.kanevsky at gmail.com (arkady kanevsky) Date: Mon, 16 Mar 2009 20:09:21 -0400 Subject: ***SPAM*** Re: [ofa-general] [ANNOUNCE] compat-dapl-1.2.13 and dapl-2.0.16 Release In-Reply-To: References: Message-ID: <517c62fb0903161709y5b52a0fasfa65b35e8acb1e58@mail.gmail.com> Arlin, Any plan to extend dapltest to RDMA write with immediate data, and other IB specific DAPL 2.0 extensions? It would be nice to add it to interop testing. Thanks, Arkady On Mon, Mar 16, 2009 at 7:41 PM, Davis, Arlin R wrote: > > New DAPL releases now available from OFA download page: > > http://www.openfabrics.org/downloads/dapl/ > > Latest Packages: > > md5sum: 74974ec74e3d4c254698af240ba5e415 compat-dapl-1.2.13.tar.gz > md5sum: 890e22b42397fac9c0c43a97507d1d4c dapl-2.0.16.tar.gz > > Vlad, please pick up new packages and install following for OFED 1.4.1 RC2: > > For 1.2 and 2.0 support on same system, including development, install RPM > packages as follow: > > compat-dapl-1.2.13-1 > compat-dapl-devel-1.2.13-1 > dapl-2.0.16-1 > dapl-utils-2.0.16-1 > dapl-devel-2.0.16-1 > dapl-debuginfo-2.0.16-1 > > Summary of changes for lastest release: > > v1,v2 - spec files missing Requires(post) statements for sed/coreutils > v1,v2 - dtest/dapltest: use $(top_builddir) for .la files during test > builds > v1,v2 - scm: remove unecessary thread when using direct objects > v2 - scm: remove query gid/lid from connection setup phase > v2 - common: add optional 64 bit counters for IA, EP, EVD's (with > -DDAPL_COUNTERS) > v2 - dtestx: add option -p to print and query new counters > v2 - WinOF and OFED: sync up trees for common code base. > > -arlin > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > -- Cheers, Arkady Kanevsky -------------- next part -------------- An HTML attachment was scrubbed... URL: From ogerlitz at voltaire.com Mon Mar 16 23:59:01 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 17 Mar 2009 08:59:01 +0200 Subject: [ofa-general] Re: [PATCH] libibmad: Add PortSelect and CounterSelect fields for PortXmit/RcvDataSL attributes In-Reply-To: <20090313203620.GA6173@comcast.net> References: <20090313203620.GA6173@comcast.net> Message-ID: <49BF4A35.5000209@voltaire.com> Hal Rosenstock wrote: > Add PortSelect and CounterSelect fields for PortXmit/RcvDataSL attributes > Can this changed be validated with ibsim? if not, do we want to allow commits that were's tested with niether ibsim nor real HW? Or. From tziporet at dev.mellanox.co.il Tue Mar 17 01:27:26 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Tue, 17 Mar 2009 10:27:26 +0200 Subject: [ofa-general] entry points in OFED kernel module In-Reply-To: References: Message-ID: <49BF5EEE.5090404@mellanox.co.il> sumeet gandhare wrote: > Hi, > I am new to OFED stack and wish to understand the design from the > source code. I have read the chapter 3 of IBA spec from OpenFabrics > and understand the concept of Queue Pair, Completion Queue, work > request, consumer, LID, GUID. Basically in IB, the consumer process > enqueues the work request to the send queue pair which specifies the > virtual address of the message to be sent and destination LID and QPN. > The receiving consumer enqueues work elements to receive queue > specifying the virtual address of the message being received. On > arrival of the message , the completion queue contains a CQE > notification to the client. So hardware basically is a DMA engine > which transfer the message from source virtual address to desitnation > virtual address. I wish to understand how this part better. > > Please could you suggest entry points in the stack like verbs and the > interaction with device driver like iPath and Mlx.. What all files I > should look at and in what sequence to better understand the > programming model and its implementation to send and receive the IB > packets on the wire? I suggest you look at examples under libibverbs directory, for writing verbs applications As for the drivers they are located under drivers/infiniband/hw/ Tziporet From Zhen.Liang at Sun.COM Tue Mar 17 02:45:20 2009 From: Zhen.Liang at Sun.COM (Liang Zhen) Date: Tue, 17 Mar 2009 17:45:20 +0800 Subject: [ofa-general] OFED 1.4's autoconf.h conflicting with kernel In-Reply-To: <1233180648.9119.32.camel@pc.interlinx.bc.ca> References: <1232736152.7440.127.camel@pc.interlinx.bc.ca> <497A141D.90008@nasa.gov> <1233180648.9119.32.camel@pc.interlinx.bc.ca> Message-ID: <49BF7130.1010902@sun.com> Hi, could somebody help us on this? We are blocked somehow by this, of couse we can remove the "autoconf.h" before we build our software depending on OFED, but I don't think it's the supposed way... Regards Liang Brian J. Murrell wrote: > On Fri, 2009-01-23 at 11:01 -0800, Jeff Becker wrote: > >> Hi Brian >> > > Hi (again) Jeff (and everyone else, especially however maintains the > packaging of /usr/src/ofa_kernel), > > >> Brian J. Murrell wrote: >> >>> Some research has led me to a message >>> (http://www.mail-archive.com/general at lists.openfabrics.org/msg18161.html) from Jeff Becker back on Thu, 10 Jul 2008 15:58:53 -0700 in which he submitted a patch to integrate NFSRDMA into OFED 1.4 which is what appears to have brought these changes into OFED 1.4. >>> > > The more I look at this, the more I'm convinced there is either an angle > I am completely missing or this is just plainly not the way to do this. > > It just cannot work to have two "linux/autoconf.h" files for a third > (where the first two parties are OFED and the kernel) party module > build. There is no guarantee that the third party module won't need to > query about various CONFIG_ definitions of both the kernel and the OFED > stack. > > The only way I can think of making this work is to "somehow" "unionize" > these two files (i.e. so there is a single "superset" of them both). > Perhaps it's doable with some kind of #include_next magic, perhaps not. > > >> I usually build my kernel first (usually with NFS). Then I build OFED. >> > > Right. This is simple enough. It's when you want to build another > kernel module that wants the OFED stack that things get sticky. > > I realize this is probably not really your area of responsibility and > this dual autoconf.h problem pre-existed your patch, but your patch has > really exacerbated the issue by directly conflicting (CONFIG_SYSCTL is > the particular example I have on hand at the moment) some of the non-IB > kernel defines. > > I'd love to engage whoever is directly responsible for this area of the > stack but nobody seems to be responding to my queries or the bug which I > filed yesterday (which is admittedly a short time ago). I'd try just > posting a patch to fix it, but I think this needs some discussion on how > to really achieve the end goal. > > Cheers, > b. > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From tziporet at dev.mellanox.co.il Tue Mar 17 02:54:15 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Tue, 17 Mar 2009 11:54:15 +0200 Subject: [ofa-general] RE: OPENIB/OFED Tutorial? In-Reply-To: <4C2744E8AD2982428C5BFE523DF8CDCB43E6D5A502@MNEXMB1.qlogic.org> References: <4C2744E8AD2982428C5BFE523DF8CDCB43E6D5A4DC@MNEXMB1.qlogic.org> <4C2744E8AD2982428C5BFE523DF8CDCB43E6D5A502@MNEXMB1.qlogic.org> Message-ID: <49BF7347.8000600@mellanox.co.il> Mike Heinz wrote: > > To be clear, since this is meant as a teaching tool I’m looking for > some of the more low-level details like how to fill out the qp > attributes, rather than just using the rdma_cm. > We raised this request to MWG since I think OFA should take care of having such document Tziporet From dorfman.eli at gmail.com Tue Mar 17 03:16:18 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Tue, 17 Mar 2009 12:16:18 +0200 Subject: [ofa-general] ***SPAM*** Re: [PATCH v2] opensm: support routing engine update In-Reply-To: <49BE0CC4.6030600@gmail.com> References: <49A6B618.1090300@gmail.com> <49A6B6EB.80700@gmail.com> <20090312132137.GB8818@sashak.voltaire.com> <49B92C27.7060904@gmail.com> <20090312160528.GW8818@sashak.voltaire.com> <49BE0CC4.6030600@gmail.com> Message-ID: <49BF7872.6060102@gmail.com> This patch fixes a bug in "support routing engine update" patch. (Do you want a new patch that includes both?) It fixes cleanup of last routing engine Also use routing engine context instead of initialized flag. Signed-off-by: Eli Dorfman --- opensm/include/opensm/osm_opensm.h | 1 - opensm/opensm/osm_opensm.c | 1 - opensm/opensm/osm_ucast_mgr.c | 7 ++++--- 3 files changed, 4 insertions(+), 5 deletions(-) diff --git a/opensm/include/opensm/osm_opensm.h b/opensm/include/opensm/osm_opensm.h index d297dd8..401a73f 100644 --- a/opensm/include/opensm/osm_opensm.h +++ b/opensm/include/opensm/osm_opensm.h @@ -122,7 +122,6 @@ typedef enum _osm_routing_engine_type { struct osm_routing_engine { const char *name; void *context; - int initialized; int (*setup) (void *re, void *p_osm); int (*build_lid_matrices) (void *context); int (*ucast_build_fwd_tables) (void *context); diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c index 732f427..a9908e3 100644 --- a/opensm/opensm/osm_opensm.c +++ b/opensm/opensm/osm_opensm.c @@ -237,7 +237,6 @@ static void update_routing_engine( struct osm_routing_engine *last) { cur->context = last->context; - cur->initialized = last->initialized; cur->setup = last->setup; cur->build_lid_matrices = last->build_lid_matrices; cur->ucast_build_fwd_tables = last->ucast_build_fwd_tables; diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c index abae978..349c4bb 100644 --- a/opensm/opensm/osm_ucast_mgr.c +++ b/opensm/opensm/osm_ucast_mgr.c @@ -970,7 +970,7 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) p_osm->routing_engine_used = OSM_ROUTING_ENGINE_TYPE_NONE; while (p_routing_eng) { - if (!p_routing_eng->initialized && + if (!p_routing_eng->context && p_routing_eng->setup(p_routing_eng, p_osm)) { OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A0F: setup of routing engine \'%s\' failed\n", @@ -980,7 +980,6 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) } OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, "\'%s\' routing engine set up\n", p_routing_eng->name); - p_routing_eng->initialized = 1; if (!ucast_mgr_route(p_routing_eng, p_osm)) break; @@ -989,6 +988,7 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) if (p_routing_eng->delete) p_routing_eng->delete(p_routing_eng->context); + p_routing_eng->context = NULL; p_routing_eng = p_routing_eng->next; } @@ -1003,12 +1003,13 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) /* cleanup last unused routing engine */ p_routing_eng = p_osm->last_routing_engine; if (p_routing_eng) { - if (p_routing_eng->initialized && + if (p_routing_eng->context && p_routing_eng->delete && p_osm->routing_engine_used != osm_routing_engine_type(p_routing_eng->name)) p_routing_eng->delete(p_routing_eng->context); free(p_routing_eng); + p_osm->last_routing_engine = NULL; } OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, -- 1.5.5 From vlad at lists.openfabrics.org Tue Mar 17 03:53:31 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Tue, 17 Mar 2009 03:53:31 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090317-0200 daily build status Message-ID: <20090317105331.DBEA8E60968@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.27 Passed on i686 with linux-2.6.26 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: Build failed on x86_64 with linux-2.6.18-128.el5 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Wstrict-prototypes -Wundef -Werror-implicit-function-declaration -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(af_rds)" -D"KBUILD_MODNAME=KBUILD_STR(rds)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check/net/rds/.tmp_af_rds.o /home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check/net/rds/af_rds.c In file included from /home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check/net/rds/af_rds.c:41: /home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check/net/rds/rds.h:12: error: redefinition of typedef '__sum16' /home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check/kernel_addons/backport/2.6.18-EL5.3/include/linux/types.h:6: error: previous declaration of '__sum16' was here make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check/net/rds/af_rds.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check/net/rds] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From hal.rosenstock at gmail.com Tue Mar 17 04:00:00 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 17 Mar 2009 07:00:00 -0400 Subject: [ofa-general] Re: [PATCH] libibmad: Add PortSelect and CounterSelect fields for PortXmit/RcvDataSL attributes In-Reply-To: <49BF4A35.5000209@voltaire.com> References: <20090313203620.GA6173@comcast.net> <49BF4A35.5000209@voltaire.com> Message-ID: On Tue, Mar 17, 2009 at 2:59 AM, Or Gerlitz wrote: > Hal Rosenstock wrote: >> >> Add PortSelect and CounterSelect fields for PortXmit/RcvDataSL attributes >> > > Can this changed be validated with ibsim? Yes; I have changes to ibsim which I haven't yet submitted. Also, is there any risk with these changes ? > if not, do we want to allow > commits that were's tested with niether ibsim nor real HW? There are no changes I've submitted which haven't been tested with either ibsim or real HW. -- Hal > Or. > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From sashak at voltaire.com Tue Mar 17 05:36:38 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Mar 2009 14:36:38 +0200 Subject: [ofa-general] Re: [PATCH] opensm/osm_link_mgr.c Remove __osm_ prefix In-Reply-To: <49BE4350.4000001@Sun.COM> References: <49BE4350.4000001@Sun.COM> Message-ID: <20090317123638.GG12557@sashak.voltaire.com> On 13:17 Mon 16 Mar , Line.Holen at Sun.COM wrote: > > Remove all __osm_ prefixes from static functions > > Signed-off-by: Line Holen Applied. Thanks. Sasha From sashak at voltaire.com Tue Mar 17 05:37:06 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Mar 2009 14:37:06 +0200 Subject: [ofa-general] Re: [PATCH TRIVIAL] opensm/osm_opensm.c: add newline to log message In-Reply-To: <49BE0ED5.3030702@gmail.com> References: <49BE0ED5.3030702@gmail.com> Message-ID: <20090317123706.GH12557@sashak.voltaire.com> On 10:33 Mon 16 Mar , Eli Dorfman (Voltaire) wrote: > add newline to log message > > Signed-off-by: Eli Dorfman Aplied. Thanks. Sasha From sashak at voltaire.com Tue Mar 17 05:41:44 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Mar 2009 14:41:44 +0200 Subject: [ofa-general] Re: [PATCH] opensm/osm_req.c: Update log message based on commit 3551389dcb7353ffd51c66e6ad518648bc1dd19e In-Reply-To: <20090316123935.GA26174@comcast.net> References: <20090316123935.GA26174@comcast.net> Message-ID: <20090317124137.GI12557@sashak.voltaire.com> On 07:39 Mon 16 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Aplied. Thanks. Sasha From sashak at voltaire.com Tue Mar 17 05:42:22 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Mar 2009 14:42:22 +0200 Subject: [ofa-general] Re: [PATCH] libibmad/libibmad.map: Eliminate perf_classportinfo_query_via In-Reply-To: <20090316165225.GA7793@comcast.net> References: <20090316165225.GA7793@comcast.net> Message-ID: <20090317124222.GJ12557@sashak.voltaire.com> On 11:52 Mon 16 Mar , Hal Rosenstock wrote: > > since it was eliminated by commit 6961ab9e34332f88a2688866b37e6268f9d587f6 > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Tue Mar 17 05:48:11 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Mar 2009 14:48:11 +0200 Subject: [ofa-general] Re: [PATCH v2] opensm: support routing engine update In-Reply-To: <49BF7872.6060102@gmail.com> References: <49A6B618.1090300@gmail.com> <49A6B6EB.80700@gmail.com> <20090312132137.GB8818@sashak.voltaire.com> <49B92C27.7060904@gmail.com> <20090312160528.GW8818@sashak.voltaire.com> <49BE0CC4.6030600@gmail.com> <49BF7872.6060102@gmail.com> Message-ID: <20090317124803.GK12557@sashak.voltaire.com> On 12:16 Tue 17 Mar , Eli Dorfman (Voltaire) wrote: > This patch fixes a bug in "support routing engine update" patch. > (Do you want a new patch that includes both?) Yes. But wait a little with V3 posting - I have a comments for the current (V2) version. > It fixes cleanup of last routing engine > Also use routing engine context instead of initialized flag. > > Signed-off-by: Eli Dorfman > --- > opensm/include/opensm/osm_opensm.h | 1 - > opensm/opensm/osm_opensm.c | 1 - > opensm/opensm/osm_ucast_mgr.c | 7 ++++--- > 3 files changed, 4 insertions(+), 5 deletions(-) > > diff --git a/opensm/include/opensm/osm_opensm.h b/opensm/include/opensm/osm_opensm.h > index d297dd8..401a73f 100644 > --- a/opensm/include/opensm/osm_opensm.h > +++ b/opensm/include/opensm/osm_opensm.h > @@ -122,7 +122,6 @@ typedef enum _osm_routing_engine_type { > struct osm_routing_engine { > const char *name; > void *context; > - int initialized; Assumption that routing engine setup will always set context value can be not correct. Sasha From tziporet at mellanox.co.il Tue Mar 17 05:52:13 2009 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Tue, 17 Mar 2009 14:52:13 +0200 Subject: [ofa-general] OFED 1.4.1 RC2 build update Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD0213CF5A@mtlexch01.mtl.com> We plan to build OFED 1.4.1 tomorrow All - please send all your commits to Vlad TODAY If someone has an issue with this plan please let me know thanks Tziporet -------------- next part -------------- An HTML attachment was scrubbed... URL: From sashak at voltaire.com Tue Mar 17 06:35:48 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Mar 2009 15:35:48 +0200 Subject: [ofa-general] Re: [PATCH v2] opensm: support routing engine update In-Reply-To: <49BE0CC4.6030600@gmail.com> References: <49A6B618.1090300@gmail.com> <49A6B6EB.80700@gmail.com> <20090312132137.GB8818@sashak.voltaire.com> <49B92C27.7060904@gmail.com> <20090312160528.GW8818@sashak.voltaire.com> <49BE0CC4.6030600@gmail.com> Message-ID: <20090317133548.GL12557@sashak.voltaire.com> On 10:24 Mon 16 Mar , Eli Dorfman (Voltaire) wrote: > diff --git a/opensm/include/opensm/osm_opensm.h b/opensm/include/opensm/osm_opensm.h > index c121be4..d297dd8 100644 > --- a/opensm/include/opensm/osm_opensm.h > +++ b/opensm/include/opensm/osm_opensm.h > @@ -122,6 +122,8 @@ typedef enum _osm_routing_engine_type { > struct osm_routing_engine { > const char *name; > void *context; > + int initialized; > + int (*setup) (void *re, void *p_osm); In order to avoid ugly castings here you need to declare a real function prototype: int (*)(struct osm_routing_engine *, struct osm_opensm *); For this you probably should put type forward declarations above those lines: struct osm_routing_engine; struct osm_opensm; > int (*build_lid_matrices) (void *context); > int (*ucast_build_fwd_tables) (void *context); > void (*ucast_dump_tables) (void *context); > @@ -183,6 +185,7 @@ typedef struct osm_opensm { > cl_dispatcher_t disp; > cl_plock_t lock; > struct osm_routing_engine *routing_engine_list; > + struct osm_routing_engine *last_routing_engine; > osm_routing_engine_type_t routing_engine_used; > osm_stats_t stats; > osm_console_t console; > @@ -523,5 +526,7 @@ extern volatile unsigned int osm_exit_flag; > * Set to one to cause all threads to leave > *********/ > > +void osm_update_routing_engines(osm_opensm_t *osm, const char *engine_names); > + > END_C_DECLS > #endif /* _OSM_OPENSM_H_ */ > diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c > index cfe6474..7126658 100644 > --- a/opensm/opensm/osm_opensm.c > +++ b/opensm/opensm/osm_opensm.c > @@ -169,14 +169,7 @@ static void setup_routing_engine(osm_opensm_t *osm, const char *name) > memset(re, 0, sizeof(struct osm_routing_engine)); > > re->name = m->name; > - if (m->setup(re, osm)) { > - OSM_LOG(&osm->log, OSM_LOG_VERBOSE, > - "setup of routing" > - " engine \'%s\' failed\n", name); > - return; > - } > - OSM_LOG(&osm->log, OSM_LOG_DEBUG, > - "\'%s\' routing engine set up\n", re->name); > + re->setup = (int (*)(void *, void *))m->setup; > append_routing_engine(osm, re); > return; > } > @@ -236,6 +229,55 @@ static void destroy_routing_engines(osm_opensm_t *osm) > r->delete(r->context); > free(r); > } > + osm->routing_engine_list = NULL; > +} > + > +static void update_routing_engine( > + struct osm_routing_engine *cur, > + struct osm_routing_engine *last) > +{ > + cur->context = last->context; > + cur->initialized = last->initialized; > + cur->setup = last->setup; > + cur->build_lid_matrices = last->build_lid_matrices; > + cur->ucast_build_fwd_tables = last->ucast_build_fwd_tables; > + cur->ucast_dump_tables = last->ucast_dump_tables; > + cur->delete = last->delete; > +} This is memcpy(), isn't it? > + > +void osm_update_routing_engines(osm_opensm_t *osm, const char *engine_names) > +{ > + struct osm_routing_engine *r, *l; > + > + /* find used routing engine and save as last */ > + l = r = osm->routing_engine_list; > + if (r && osm->routing_engine_used == osm_routing_engine_type(r->name)) { > + osm->last_routing_engine = r; > + osm->routing_engine_list = r->next; > + } > + else while ((r = r->next)) { > + if (osm->routing_engine_used == > + osm_routing_engine_type(r->name)) { > + osm->last_routing_engine = r; > + l->next = r->next; > + break; > + } > + l = r; > + } > + /* cleanup prev routing engine list and replace with current list */ > + destroy_routing_engines(osm); > + setup_routing_engines(osm, engine_names); > + > + /* check if last routing engine exist in new list and update callbacks */ > + r = osm->routing_engine_list; > + while (r) { > + if (osm->routing_engine_used == > + osm_routing_engine_type(r->name)) { > + update_routing_engine(r, osm->last_routing_engine); > + break; > + } > + r = r->next; > + } > } > > /********************************************************************** > diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c > index fe0a446..abae978 100644 > --- a/opensm/opensm/osm_ucast_mgr.c > +++ b/opensm/opensm/osm_ucast_mgr.c > @@ -970,8 +970,25 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) > > p_osm->routing_engine_used = OSM_ROUTING_ENGINE_TYPE_NONE; > while (p_routing_eng) { > + if (!p_routing_eng->initialized && > + p_routing_eng->setup(p_routing_eng, p_osm)) { > + OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, > + "ERR 3A0F: setup of routing engine \'%s\' failed\n", > + p_routing_eng->name); > + p_routing_eng = p_routing_eng->next; > + continue; > + } > + OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, > + "\'%s\' routing engine set up\n", p_routing_eng->name); > + p_routing_eng->initialized = 1; > + > if (!ucast_mgr_route(p_routing_eng, p_osm)) > break; > + > + /* delete unused routing engine */ > + if (p_routing_eng->delete) > + p_routing_eng->delete(p_routing_eng->context); > + Now it can be deleted twice - here and in destroy_routing_engines()? Also shouldn't you drop initialized flag? Sasha From brian at sun.com Tue Mar 17 07:12:13 2009 From: brian at sun.com (Brian J. Murrell) Date: Tue, 17 Mar 2009 10:12:13 -0400 Subject: [ofa-general] OFED 1.4's autoconf.h conflicting with kernel In-Reply-To: <49BF7130.1010902@sun.com> References: <1232736152.7440.127.camel@pc.interlinx.bc.ca> <497A141D.90008@nasa.gov> <1233180648.9119.32.camel@pc.interlinx.bc.ca> <49BF7130.1010902@sun.com> Message-ID: <1237299133.3333.8.camel@pc.interlinx.bc.ca> On Tue, 2009-03-17 at 17:45 +0800, Liang Zhen wrote: > Hi, could somebody help us on this? We are blocked somehow by this, Liang, We are not so much "blocked" as much as it's just a big PITA because until now, we have not used gcc's "-include" to get the include/linux/autoconf.h from the kernel into our modules, but rather used #include in the .c (and .h) files where we needed it. We could go undo all of that (using #include) and using gcc's "-include" to get them both included in order of preference, but as I say, it's a lot of code to change, in both our .c (and .h) files as well as all of our autoconf macros. I have an open bug on this at https://bugs.openfabrics.org/show_bug.cgi?id=1538 if you want subscribe to it and follow along. Cheers, b. -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From itaib at mellanox.com Tue Mar 17 07:14:41 2009 From: itaib at mellanox.com (Itai Baz) Date: Tue, 17 Mar 2009 16:14:41 +0200 Subject: [ofa-general] [PATCH] ib_types.h: Adding BKEY violation trap (259) Message-ID: <200903171614.41707.itaib@mellanox.com> Hi Sasha, The following patch adds B_Key violation trap definition. Signed-off-by: Itai Baz --- opensm/include/iba/ib_types.h | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-) diff --git a/opensm/include/iba/ib_types.h b/opensm/include/iba/ib_types.h index e7cbb79..eac85f3 100644 --- a/opensm/include/iba/ib_types.h +++ b/opensm/include/iba/ib_types.h @@ -7162,6 +7162,17 @@ typedef struct _ib_mad_notice_attr // Total Size calc Accumulated ib_net16_t sw_lid; // 2 uint8_t port_no; // 1 } PACK_SUFFIX ntc_259; + struct _ntc_bkey_259 // bkey violation + { + ib_net16_t lidaddr; + uint8_t method; + uint8_t reserved; + ib_net16_t attribute_id; + ib_net32_t attribute_modifier; + ib_net32_t qp; // qp is low 24 bits + ib_net64_t bkey; + ib_gid_t gid; + } PACK_SUFFIX ntc_bkey_259; } data_details; ib_gid_t issuer_gid; // 16 80 } PACK_SUFFIX ib_mad_notice_attr_t; -- 1.5.3.7 From itaib at mellanox.com Tue Mar 17 07:17:56 2009 From: itaib at mellanox.com (Itai Baz) Date: Tue, 17 Mar 2009 16:17:56 +0200 Subject: [ofa-general] [PATCH] libibmad/serv.c: Fixed respond function to return proper result code Message-ID: <200903171617.56184.itaib@mellanox.com> Hi Sasha, mad_respond_via() function was always returning OK. Fixing it to return the proper status. Signed-off-by: Itai Baz --- libibmad/src/serv.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/libibmad/src/serv.c b/libibmad/src/serv.c index c97466a..7e93c12 100644 --- a/libibmad/src/serv.c +++ b/libibmad/src/serv.c @@ -128,6 +128,7 @@ int mad_respond_via(void *umad, ib_portid_t * portid, uint32_t rstatus, rpc.oui = mad_get_field(mad, 0, IB_VEND2_OUI_F); rpc.trid = mad_get_field64(mad, 0, IB_MAD_TRID_F); + rpc.rstatus = rstatus; /* cleared by default: timeout, datasz, dataoffs, mkey, mask */ -- 1.5.3.7 From itaib at mellanox.com Tue Mar 17 07:22:34 2009 From: itaib at mellanox.com (Itai Baz) Date: Tue, 17 Mar 2009 16:22:34 +0200 Subject: [ofa-general] [PATCH] libibmad: added support for handling of BM (Baseboard management) MADs Message-ID: <200903171622.34568.itaib@mellanox.com> Hi Sasha, This patch adds support for handling of BM (Baseboard management) MADs. Signed-off-by: Itai Baz --- libibmad/Makefile.am | 3 +- libibmad/include/infiniband/mad.h | 49 ++++++++++++++++++ libibmad/src/bm.c | 101 +++++++++++++++++++++++++++++++++++++ libibmad/src/libibmad.map | 1 + libibmad/src/register.c | 2 + 5 files changed, 155 insertions(+), 1 deletions(-) create mode 100644 libibmad/src/bm.c diff --git a/libibmad/Makefile.am b/libibmad/Makefile.am index 4f3ba98..618ed3e 100644 --- a/libibmad/Makefile.am +++ b/libibmad/Makefile.am @@ -15,7 +15,8 @@ endif libibmad_la_SOURCES = src/dump.c src/fields.c src/mad.c src/portid.c \ src/resolve.c src/rpc.c src/sa.c src/smp.c src/gs.c \ - src/serv.c src/register.c src/vendor.c src/mad_internal.h + src/serv.c src/register.c src/vendor.c src/mad_internal.h \ + src/bm.c libibmad_la_LDFLAGS = -version-info $(ibmad_api_version) \ -export-dynamic $(libibmad_version_script) diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h index b8290a7..efc887a 100644 --- a/libibmad/include/infiniband/mad.h +++ b/libibmad/include/infiniband/mad.h @@ -63,6 +63,11 @@ BEGIN_C_DECLS #define IB_SA_MCM_RECSZ 53 #define IB_SA_PR_RECSZ 64 +#define IB_BM_DATA_OFFS 64 +#define IB_BM_DATA_SZ (IB_MAD_SIZE - IB_BM_DATA_OFFS) +#define IB_BM_BKEY_OFFS 24 +#define IB_BM_BKEY_AND_DATA_SZ (IB_MAD_SIZE - IB_BM_BKEY_OFFS) + enum MAD_CLASSES { IB_SMI_CLASS = 0x1, IB_SMI_DIRECT_CLASS = 0x81, @@ -108,6 +113,14 @@ enum MAD_ATTR_ID { INFORM_INFO = 0x3, }; +enum MAD_STATUS { + IB_MAD_STS_OK = (0 << 2), + IB_MAD_STS_BAD_BASE_VER_OR_CLASS = (1 << 2), + IB_MAD_STS_METHOD_NOT_SUPPORTED = (2 << 2), + IB_MAD_STS_METHOD_ATTR_NOT_SUPPORTED = (3 << 2), + IB_MAD_STS_INV_ATTR_VALUE = (7 << 2), +}; + enum SMI_ATTR_ID { IB_ATTR_NODE_DESC = 0x10, IB_ATTR_NODE_INFO = 0x11, @@ -158,6 +171,29 @@ enum GSI_ATTR_ID { IB_GSI_ATTR_LAST }; +enum BM_ATTR_ID { + IB_BM_ATTR_BKEYINFO = 0x10, + IB_BM_ATTR_WRITE_VPD = 0x20, + IB_BM_ATTR_READ_VPD = 0x21, + IB_BM_ATTR_RESET_IBML = 0x22, + IB_BM_ATTR_SET_MODULE_PM_CONTROL = 0x23, + IB_BM_ATTR_GET_MODULE_PM_CONTROL = 0x24, + IB_BM_ATTR_SET_UNIT_PM_CONTROL = 0x25, + IB_BM_ATTR_GET_UNIT_PM_CONTROL = 0x26, + IB_BM_ATTR_SET_IOC_PM_CONTROL = 0x27, + IB_BM_ATTR_GET_IOC_PM_CONTROL = 0x28, + IB_BM_ATTR_SET_MODULE_STATE = 0x29, + IB_BM_ATTR_SET_MODULE_ATTENTION = 0x2A, + IB_BM_ATTR_GET_MODULE_STATUS = 0x2B, + IB_BM_ATTR_IB2IBML = 0x2C, + IB_BM_ATTR_IB2CME = 0x2D, + IB_BM_ATTR_IB2MME = 0x2E, + IB_BM_ATTR_OEM = 0x2F, + + IB_BM_ATTR_LAST +}; + + #define IB_VENDOR_OPENIB_PING_CLASS (IB_VENDOR_RANGE2_START_CLASS + 2) #define IB_VENDOR_OPENIB_SYSSTAT_CLASS (IB_VENDOR_RANGE2_START_CLASS + 3) #define IB_OPENIB_OUI (0x001405) @@ -661,6 +697,16 @@ typedef struct ib_vendor_call { ib_rmpp_hdr_t rmpp; } ib_vendor_call_t; +typedef struct ib_bm_call { + unsigned method; + unsigned attrid; + unsigned mod; + unsigned timeout; + ib_rmpp_hdr_t rmpp; + uint64_t bkey; +} ib_bm_call_t; + + #define IB_MIN_UCAST_LID 1 #define IB_MAX_UCAST_LID (0xc000-1) #define IB_MIN_MCAST_LID 0xc000 @@ -876,6 +922,9 @@ MAD_EXPORT uint8_t *performance_reset_via(void *rcvbuf, ib_portid_t * dest, unsigned timeout, unsigned id, const struct ibmad_port *srcport); +/* bm.c */ +uint8_t * bm_rpc_call_via(void *data, ib_portid_t *portid, ib_bm_call_t *call, struct ibmad_port *srcport); + /* dump.c */ MAD_EXPORT ib_mad_dump_fn mad_dump_int, mad_dump_uint, mad_dump_hex, mad_dump_rhex, diff --git a/libibmad/src/bm.c b/libibmad/src/bm.c new file mode 100644 index 0000000..a095b9a --- /dev/null +++ b/libibmad/src/bm.c @@ -0,0 +1,101 @@ +/* + * Copyright (c) 2004-2007 Voltaire Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ + +#if HAVE_CONFIG_H +# include +#endif /* HAVE_CONFIG_H */ + +#include +#include +#include +#include +#include +#include + +#include + +#undef DEBUG +#define DEBUG if (ibdebug) IBWARN + +static inline int +response_expected(int method) +{ + return method == IB_MAD_METHOD_GET || + method == IB_MAD_METHOD_SET || + method == IB_MAD_METHOD_TRAP; +} + +uint8_t * +bm_rpc_call_via(void *data, ib_portid_t *portid, ib_bm_call_t *call, struct ibmad_port *srcport) +{ + ib_rpc_t rpc = {0}; + int resp_expected; + char data_with_bkey[IB_BM_BKEY_AND_DATA_SZ] = {0}; + + DEBUG("route %s data %p", portid2str(portid), data); + if (portid->lid <= 0) { + IBWARN("only lid routes are supported"); + return 0; + } + + resp_expected = response_expected(call->method); + + rpc.mgtclass = IB_BOARD_MGMT_CLASS; + + rpc.method = call->method; + rpc.attr.id = call->attrid; + rpc.attr.mod = call->mod; + rpc.timeout = resp_expected ? call->timeout : 0; + // send data and bkey + rpc.datasz = IB_BM_BKEY_AND_DATA_SZ; + rpc.dataoffs = IB_BM_BKEY_OFFS; + + // copy data to a buffer which also includes the bkey + *((uint64_t *) data_with_bkey) = call->bkey; + memcpy(data_with_bkey + IB_BM_DATA_OFFS - IB_BM_BKEY_OFFS, data, IB_BM_DATA_SZ); + + DEBUG("method 0x%x attr 0x%x mod 0x%x datasz %d off %d res_ex %d bkey 0x%08x%08x", + rpc.method, rpc.attr.id, rpc.attr.mod, + rpc.datasz, rpc.dataoffs, resp_expected, + (int) (call->bkey >> 32), (int) call->bkey); + + portid->qp = 1; + if (!portid->qkey) + portid->qkey = IB_DEFAULT_QP1_QKEY; + + if (resp_expected) + return mad_rpc_rmpp(srcport, &rpc, portid, 0, data_with_bkey); /* FIXME: no RMPP for now */ + + return mad_send_via(&rpc, portid, 0, data_with_bkey, srcport) < 0 ? 0 : data_with_bkey; /* FIXME: no RMPP for now */ +} + diff --git a/libibmad/src/libibmad.map b/libibmad/src/libibmad.map index 4306dbc..a08b7b2 100644 --- a/libibmad/src/libibmad.map +++ b/libibmad/src/libibmad.map @@ -102,5 +102,6 @@ IBMAD_1.3 { ib_resolve_guid_via; ib_resolve_portid_str_via; ib_resolve_self_via; + bm_rpc_call_via; local: *; }; diff --git a/libibmad/src/register.c b/libibmad/src/register.c index eac8f5d..76567e1 100644 --- a/libibmad/src/register.c +++ b/libibmad/src/register.c @@ -68,6 +68,8 @@ static int mgmt_class_vers(int mgmt_class) return 1; case IB_CC_CLASS: return 2; + case IB_BOARD_MGMT_CLASS: + return 1; } return 0; -- 1.5.3.7 From dorfman.eli at gmail.com Tue Mar 17 07:39:23 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Tue, 17 Mar 2009 16:39:23 +0200 Subject: [ofa-general] Re: [PATCH v2] opensm: support routing engine update In-Reply-To: <20090317133548.GL12557@sashak.voltaire.com> References: <49A6B618.1090300@gmail.com> <49A6B6EB.80700@gmail.com> <20090312132137.GB8818@sashak.voltaire.com> <49B92C27.7060904@gmail.com> <20090312160528.GW8818@sashak.voltaire.com> <49BE0CC4.6030600@gmail.com> <20090317133548.GL12557@sashak.voltaire.com> Message-ID: <49BFB61B.5060303@gmail.com> Sasha Khapyorsky wrote: > On 10:24 Mon 16 Mar , Eli Dorfman (Voltaire) wrote: > > >> diff --git a/opensm/include/opensm/osm_opensm.h b/opensm/include/opensm/osm_opensm.h >> index c121be4..d297dd8 100644 >> --- a/opensm/include/opensm/osm_opensm.h >> +++ b/opensm/include/opensm/osm_opensm.h >> @@ -122,6 +122,8 @@ typedef enum _osm_routing_engine_type { >> struct osm_routing_engine { >> const char *name; >> void *context; >> + int initialized; >> + int (*setup) (void *re, void *p_osm); > > In order to avoid ugly castings here you need to declare a real function > prototype: > > int (*)(struct osm_routing_engine *, struct osm_opensm *); > > For this you probably should put type forward declarations above those > lines: > > struct osm_routing_engine; > struct osm_opensm; > >> int (*build_lid_matrices) (void *context); >> int (*ucast_build_fwd_tables) (void *context); >> void (*ucast_dump_tables) (void *context); >> @@ -183,6 +185,7 @@ typedef struct osm_opensm { >> cl_dispatcher_t disp; >> cl_plock_t lock; >> struct osm_routing_engine *routing_engine_list; >> + struct osm_routing_engine *last_routing_engine; >> osm_routing_engine_type_t routing_engine_used; >> osm_stats_t stats; >> osm_console_t console; >> @@ -523,5 +526,7 @@ extern volatile unsigned int osm_exit_flag; >> * Set to one to cause all threads to leave >> *********/ >> >> +void osm_update_routing_engines(osm_opensm_t *osm, const char *engine_names); >> + >> END_C_DECLS >> #endif /* _OSM_OPENSM_H_ */ >> diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c >> index cfe6474..7126658 100644 >> --- a/opensm/opensm/osm_opensm.c >> +++ b/opensm/opensm/osm_opensm.c >> @@ -169,14 +169,7 @@ static void setup_routing_engine(osm_opensm_t *osm, const char *name) >> memset(re, 0, sizeof(struct osm_routing_engine)); >> >> re->name = m->name; >> - if (m->setup(re, osm)) { >> - OSM_LOG(&osm->log, OSM_LOG_VERBOSE, >> - "setup of routing" >> - " engine \'%s\' failed\n", name); >> - return; >> - } >> - OSM_LOG(&osm->log, OSM_LOG_DEBUG, >> - "\'%s\' routing engine set up\n", re->name); >> + re->setup = (int (*)(void *, void *))m->setup; >> append_routing_engine(osm, re); >> return; >> } >> @@ -236,6 +229,55 @@ static void destroy_routing_engines(osm_opensm_t *osm) >> r->delete(r->context); >> free(r); >> } >> + osm->routing_engine_list = NULL; >> +} >> + >> +static void update_routing_engine( >> + struct osm_routing_engine *cur, >> + struct osm_routing_engine *last) >> +{ >> + cur->context = last->context; >> + cur->initialized = last->initialized; >> + cur->setup = last->setup; >> + cur->build_lid_matrices = last->build_lid_matrices; >> + cur->ucast_build_fwd_tables = last->ucast_build_fwd_tables; >> + cur->ucast_dump_tables = last->ucast_dump_tables; >> + cur->delete = last->delete; >> +} > > This is memcpy(), isn't it? No since i don't want the next pointer. > >> + >> +void osm_update_routing_engines(osm_opensm_t *osm, const char *engine_names) >> +{ >> + struct osm_routing_engine *r, *l; >> + >> + /* find used routing engine and save as last */ >> + l = r = osm->routing_engine_list; >> + if (r && osm->routing_engine_used == osm_routing_engine_type(r->name)) { >> + osm->last_routing_engine = r; >> + osm->routing_engine_list = r->next; >> + } >> + else while ((r = r->next)) { >> + if (osm->routing_engine_used == >> + osm_routing_engine_type(r->name)) { >> + osm->last_routing_engine = r; >> + l->next = r->next; >> + break; >> + } >> + l = r; >> + } >> + /* cleanup prev routing engine list and replace with current list */ >> + destroy_routing_engines(osm); >> + setup_routing_engines(osm, engine_names); >> + >> + /* check if last routing engine exist in new list and update callbacks */ >> + r = osm->routing_engine_list; >> + while (r) { >> + if (osm->routing_engine_used == >> + osm_routing_engine_type(r->name)) { >> + update_routing_engine(r, osm->last_routing_engine); >> + break; >> + } >> + r = r->next; >> + } >> } >> >> /********************************************************************** >> diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c >> index fe0a446..abae978 100644 >> --- a/opensm/opensm/osm_ucast_mgr.c >> +++ b/opensm/opensm/osm_ucast_mgr.c >> @@ -970,8 +970,25 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) >> >> p_osm->routing_engine_used = OSM_ROUTING_ENGINE_TYPE_NONE; >> while (p_routing_eng) { >> + if (!p_routing_eng->initialized && >> + p_routing_eng->setup(p_routing_eng, p_osm)) { >> + OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, >> + "ERR 3A0F: setup of routing engine \'%s\' failed\n", >> + p_routing_eng->name); >> + p_routing_eng = p_routing_eng->next; >> + continue; >> + } >> + OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, >> + "\'%s\' routing engine set up\n", p_routing_eng->name); >> + p_routing_eng->initialized = 1; >> + >> if (!ucast_mgr_route(p_routing_eng, p_osm)) >> break; >> + >> + /* delete unused routing engine */ >> + if (p_routing_eng->delete) >> + p_routing_eng->delete(p_routing_eng->context); >> + > > Now it can be deleted twice - here and in destroy_routing_engines()? > Also shouldn't you drop initialized flag? yes, i should. the second patch fixes that. From jon at opengridcomputing.com Tue Mar 17 07:48:53 2009 From: jon at opengridcomputing.com (Jon Mason) Date: Tue, 17 Mar 2009 09:48:53 -0500 Subject: [ofa-general] ofa_1_4_kernel 20090317-0200 daily build status In-Reply-To: <20090317105331.DBEA8E60968@openfabrics.org> References: <20090317105331.DBEA8E60968@openfabrics.org> Message-ID: <20090317144852.GA24177@opengridcomputing.com> Hey Vlad, __sum16 is now defined in types.h. If you remove kernel_patches/backport/2.6.18-EL5.3/rds_to_rhel5.3.patch then the issue below will be solved. Thanks, Jon On Tue, Mar 17, 2009 at 03:53:31AM -0700, Vladimir Sokolovsky (Mellanox) wrote: > This email was generated automatically, please do not reply > > > git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git > git_branch: ofed_kernel > > Common build parameters: > > Passed: > Passed on i686 with linux-2.6.16 > Passed on i686 with linux-2.6.18 > Passed on i686 with linux-2.6.17 > Passed on i686 with linux-2.6.19 > Passed on i686 with linux-2.6.21.1 > Passed on i686 with linux-2.6.24 > Passed on i686 with linux-2.6.22 > Passed on i686 with linux-2.6.27 > Passed on i686 with linux-2.6.26 > Passed on x86_64 with linux-2.6.16 > Passed on x86_64 with linux-2.6.16.43-0.3-smp > Passed on x86_64 with linux-2.6.16.21-0.8-smp > Passed on x86_64 with linux-2.6.18 > Passed on x86_64 with linux-2.6.17 > Passed on x86_64 with linux-2.6.16.60-0.21-smp > Passed on x86_64 with linux-2.6.18-1.2798.fc6 > Passed on x86_64 with linux-2.6.18-53.el5 > Passed on x86_64 with linux-2.6.19 > Passed on x86_64 with linux-2.6.18-8.el5 > Passed on x86_64 with linux-2.6.18-93.el5 > Passed on x86_64 with linux-2.6.22 > Passed on x86_64 with linux-2.6.21.1 > Passed on x86_64 with linux-2.6.20 > Passed on x86_64 with linux-2.6.25 > Passed on x86_64 with linux-2.6.24 > Passed on x86_64 with linux-2.6.22.5-31-default > Passed on x86_64 with linux-2.6.9-42.ELsmp > Passed on x86_64 with linux-2.6.26 > Passed on x86_64 with linux-2.6.27 > Passed on x86_64 with linux-2.6.9-78.ELsmp > Passed on x86_64 with linux-2.6.9-55.ELsmp > Passed on x86_64 with linux-2.6.9-67.ELsmp > Passed on ia64 with linux-2.6.16 > Passed on ia64 with linux-2.6.16.21-0.8-default > Passed on ia64 with linux-2.6.19 > Passed on ia64 with linux-2.6.17 > Passed on ia64 with linux-2.6.18 > Passed on ia64 with linux-2.6.21.1 > Passed on ia64 with linux-2.6.23 > Passed on ia64 with linux-2.6.22 > Passed on ia64 with linux-2.6.24 > Passed on ia64 with linux-2.6.25 > Passed on ia64 with linux-2.6.26 > Passed on ppc64 with linux-2.6.16 > Passed on ppc64 with linux-2.6.17 > Passed on ppc64 with linux-2.6.18 > Passed on ppc64 with linux-2.6.18-8.el5 > Passed on ppc64 with linux-2.6.19 > > Failed: > Build failed on x86_64 with linux-2.6.18-128.el5 > Log: > -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Wstrict-prototypes -Wundef -Werror-implicit-function-declaration -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(af_rds)" -D"KBUILD_MODNAME=KBUILD_STR(rds)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check/net/rds/.tmp_af_rds.o /home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check/net/rds/af_rds.c > In file included from /home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check/net/rds/af_rds.c:41: > /home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check/net/rds/rds.h:12: error: redefinition of typedef '__sum16' > /home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check/kernel_addons/backport/2.6.18-EL5.3/include/linux/types.h:6: error: previous declaration of '__sum16' was here > make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check/net/rds/af_rds.o] Error 1 > make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check/net/rds] Error 2 > make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090317-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 > make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' > make: *** [kernel] Error 2 > ---------------------------------------------------------------------------------- > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From jrlang at uwyo.edu Tue Mar 17 07:58:28 2009 From: jrlang at uwyo.edu (jeffrey Lang) Date: Tue, 17 Mar 2009 08:58:28 -0600 Subject: [ofa-general] Need help with Infiniband problem Message-ID: <49BFBA94.5000709@uwyo.edu> First let me say, I hope this is the right list for this email, if not please forgive me. I have a small 16 node compute cluster. The university where I work at recently opened a new Datacenter. My cluster was moved from the old Datacenter. Before the move the inifiniband was working properly, after the move the ipoib has stopped working. The cluster runs Centos 4 with all the latest updates and the Centos distributed OFED code. My plan was to update the OFED code once things had restablized. For the move, I shutdown the cluster, removed the inifiniband cables and the cluster was moved. I then reinstalled the infiniband cables (not in the same order before the move) and brought every thing back up. When i brought the cluster back up the ipoib would not work. The only message in the log file is "Mar 15 04:04:32 h2o01 kernel: ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22". The master node can see all the systems: [root at h2o01 log]# ibnodes Ca : 0x00066a0098007e99 ports 1 "h2o17 HCA-1" Ca : 0x00066a0098007e9b ports 1 "h2o18 HCA-1" Ca : 0x00066a0098007e97 ports 1 "h2o16 HCA-1" Ca : 0x00066a0098007e8c ports 1 "h2o15 HCA-1" Ca : 0x00066a0098007e94 ports 1 "h2o14 HCA-1" Ca : 0x00066a0098007e93 ports 1 "h2o13 HCA-1" Ca : 0x00066a0098007e8e ports 1 "h2o12 HCA-1" Ca : 0x00066a0098007e90 ports 1 "h2o11 HCA-1" Ca : 0x00066a0098007e98 ports 1 "h2o10 HCA-1" Ca : 0x00066a0098007e95 ports 1 "h2o09 HCA-1" Ca : 0x00066a0098007e8f ports 1 "h2o08 HCA-1" Ca : 0x00066a0098007e92 ports 1 "h2o07 HCA-1" Ca : 0x00066a0098007e8d ports 1 "h2o06 HCA-1" Ca : 0x00066a0098007e91 ports 1 "h2o05 HCA-1" Ca : 0x00066a0098007e96 ports 1 "h2ocfs HCA-1" Ca : 0x00066a0098007e9c ports 1 "h2o01 HCA-1" Switch : 0x00066a00d8000593 ports 24 "SilverStorm 9024 GUID=0x00066a00d8000593" enhanced port 0 lid 1 lmc 0 I've reset the sm on the switch, but nothing seems to work. Any ideas of where to look for whats causing the problem? jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: jrlang.vcf Type: text/x-vcard Size: 298 bytes Desc: not available URL: From landman at scalableinformatics.com Tue Mar 17 09:05:10 2009 From: landman at scalableinformatics.com (Joe Landman) Date: Tue, 17 Mar 2009 12:05:10 -0400 Subject: [ofa-general] Need help with Infiniband problem In-Reply-To: <49BFBA94.5000709@uwyo.edu> References: <49BFBA94.5000709@uwyo.edu> Message-ID: <49BFCA36.3050104@scalableinformatics.com> jeffrey Lang wrote: > First let me say, I hope this is the right list for this email, if not > please forgive me. > > I have a small 16 node compute cluster. The university where I work > at recently opened a new Datacenter. My cluster was moved from the old > Datacenter. Before the move the inifiniband was working properly, > after the move the ipoib has stopped working. [...] > I've reset the sm on the switch, but nothing seems to work. > > Any ideas of where to look for whats causing the problem? Could you do an lsmod | grep ib_ I assume you did an /etc/init.d/openibd restart If not, now is a good time ... then rerun the lsmod above. If you don't see ib_ipoib, then you might try this ifconfig ib0 up then send the output of lsmod | grep ib_ ifconfig ib0 If these still don't work, try modprobe ib_ipoib ifconfig ib0 up ifconfig ib0 -- Joseph Landman, Ph.D Founder and CEO Scalable Informatics LLC, email: landman at scalableinformatics.com web : http://www.scalableinformatics.com http://jackrabbit.scalableinformatics.com phone: +1 734 786 8423 x121 fax : +1 866 888 3112 cell : +1 734 612 4615 From jrlang at uwyo.edu Tue Mar 17 08:33:56 2009 From: jrlang at uwyo.edu (jeffrey Lang) Date: Tue, 17 Mar 2009 09:33:56 -0600 Subject: [ofa-general] Need help with Infiniband problem In-Reply-To: <49BFCA36.3050104@scalableinformatics.com> References: <49BFBA94.5000709@uwyo.edu> <49BFCA36.3050104@scalableinformatics.com> Message-ID: <49BFC2E4.5070208@uwyo.edu> As joe requested, here's before reloading the openIB stuff [root at h2o01 ~]# lsmod | grep ib_ ib_srp 38281 0 ib_sdp 54785 0 rdma_cm 39381 2 ib_sdp,rdma_ucm ib_addr 11081 1 rdma_cm ib_mthca 158357 0 ib_ipoib 96673 0 ib_umad 20969 0 ib_ucm 20937 0 ib_uverbs 43377 2 rdma_ucm,ib_ucm ib_cm 42217 4 ib_srp,rdma_cm,ib_ipoib,ib_ucm ib_sa 48841 4 ib_srp,rdma_cm,ib_ipoib,ib_cm ib_mad 43497 4 ib_mthca,ib_umad,ib_cm,ib_sa ib_core 69825 13 ib_srp,ib_sdp,rdma_ucm,rdma_cm,iw_cm,ib_mthca,ib_ipoib,ib_umad,ib_ucm,ib_uverbs,ib_cm,ib_sa,ib_mad ipv6 285729 29 ib_ipoib scsi_mod 145425 3 ib_srp,libata,sd_mod [root at h2o01 ~]# /etc/init.d/openibd restart Unloading OpenIB kernel modules: [ OK ] Loading OpenIB kernel modules: [ OK ] [root at h2o01 ~]# [root at h2o01 ~]# lsmod | grep ib_ ib_srp 38281 0 ib_sdp 54785 0 ib_ipoib 96673 0 rdma_cm 39381 2 ib_sdp,rdma_ucm ib_addr 11081 1 rdma_cm ib_mthca 158357 0 ib_umad 20969 0 ib_ucm 20937 0 ib_uverbs 43377 2 rdma_ucm,ib_ucm ib_cm 42217 4 ib_srp,ib_ipoib,rdma_cm,ib_ucm ib_sa 48841 4 ib_srp,ib_ipoib,rdma_cm,ib_cm ib_mad 43497 4 ib_mthca,ib_umad,ib_cm,ib_sa ib_core 69825 13 ib_srp,ib_sdp,ib_ipoib,rdma_ucm,rdma_cm,iw_cm,ib_mthca,ib_umad,ib_ucm,ib_uverbs,ib_cm,ib_sa,ib_mad ipv6 285729 29 ib_ipoib scsi_mod 145425 3 ib_srp,libata,sd_mod [root at h2o01 ~]# ifconfig ib0 up [root at h2o01 ~]# ifconfig ib0 ib0 Link encap:UNSPEC HWaddr 80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00 inet addr:192.168.2.1 Bcast:192.168.2.255 Mask:255.255.255.0 UP BROADCAST MULTICAST MTU:2044 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:256 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) As you can see that the ib0 interface does come up and routing seems to be setup properly Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 192.168.2.0 * 255.255.255.0 U 0 0 0 ib0 10.84.4.0 * 255.255.255.0 U 0 0 0 eth0 192.168.1.0 * 255.255.255.0 U 0 0 0 eth1 169.254.0.0 * 255.255.0.0 U 0 0 0 ib0 224.0.0.0 * 240.0.0.0 U 0 0 0 eth1 default 10.84.4.1 0.0.0.0 UG 0 0 But if i ping a node, i get nothing: [root at h2o01 ~]# ping h2oi05.cluster PING h2oi05.cluster (192.168.2.5) 56(84) bytes of data. From h2oi01.cluster (192.168.2.1) icmp_seq=0 Destination Host Unreachable From h2oi01.cluster (192.168.2.1) icmp_seq=1 Destination Host Unreachable From h2oi01.cluster (192.168.2.1) icmp_seq=2 Destination Host Unreachable From h2oi01.cluster (192.168.2.1) icmp_seq=4 Destination Host Unreachable From h2oi01.cluster (192.168.2.1) icmp_seq=5 Destination Host Unreachable From h2oi01.cluster (192.168.2.1) icmp_seq=6 Destination Host Unreachable --- h2oi05.cluster ping statistics --- 8 packets transmitted, 0 received, +6 errors, 100% packet loss, time 7000ms , pipe 4 I did ping myself and i get : [root at h2o01 ~]# ping h2oi01.cluster PING h2oi01.cluster (192.168.2.1) 56(84) bytes of data. 64 bytes from h2oi01.cluster (192.168.2.1): icmp_seq=0 ttl=64 time=0.018 ms 64 bytes from h2oi01.cluster (192.168.2.1): icmp_seq=1 ttl=64 time=0.010 ms 64 bytes from h2oi01.cluster (192.168.2.1): icmp_seq=2 ttl=64 time=0.011 ms 64 bytes from h2oi01.cluster (192.168.2.1): icmp_seq=3 ttl=64 time=0.011 ms 64 bytes from h2oi01.cluster (192.168.2.1): icmp_seq=4 ttl=64 time=0.015 ms 64 bytes from h2oi01.cluster (192.168.2.1): icmp_seq=5 ttl=64 time=0.008 ms --- h2oi01.cluster ping statistics --- 6 packets transmitted, 6 received, 0% packet loss, time 4999ms rtt min/avg/max/mdev = 0.008/0.012/0.018/0.004 ms, pipe 2 It appears that the Ip stack over IB is up and installed, just not talking on the wire or passing thru the switch. jeff Joe Landman wrote: > jeffrey Lang wrote: > >> First let me say, I hope this is the right list for this email, if not >> please forgive me. >> >> I have a small 16 node compute cluster. The university where I work >> at recently opened a new Datacenter. My cluster was moved from the old >> Datacenter. Before the move the inifiniband was working properly, >> after the move the ipoib has stopped working. >> > > [...] > > >> I've reset the sm on the switch, but nothing seems to work. >> >> Any ideas of where to look for whats causing the problem? >> > > Could you do an > > lsmod | grep ib_ > > I assume you did an > > /etc/init.d/openibd restart > > If not, now is a good time ... then rerun the lsmod above. > > If you don't see ib_ipoib, then you might try this > > ifconfig ib0 up > > then send the output of > > lsmod | grep ib_ > ifconfig ib0 > > If these still don't work, try > > modprobe ib_ipoib > ifconfig ib0 up > ifconfig ib0 > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: jrlang.vcf Type: text/x-vcard Size: 298 bytes Desc: not available URL: From yevgenyp at mellanox.co.il Tue Mar 17 08:34:43 2009 From: yevgenyp at mellanox.co.il (Yevgeny Petrilin) Date: Tue, 17 Mar 2009 17:34:43 +0200 Subject: [ofa-general][PATCH] mlx4_ib: Unregister IB device prior to performing CLOSE PORT command Message-ID: <49BFC313.1030901@mellanox.co.il> According to ConnectX PRM all operations should be stopped, all QPs should be teared down and all WQEs flushed before CLOSE_PORT command is invoked. In some cases reversed order of operations (as implemented now) could cause loss of completions. Signed-off-by: Yevgeny Petrilin --- drivers/infiniband/hw/mlx4/main.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index 61588bd..2ccb9d3 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -699,11 +699,12 @@ static void mlx4_ib_remove(struct mlx4_dev *dev, void *ibdev_ptr) struct mlx4_ib_dev *ibdev = ibdev_ptr; int p; + mlx4_ib_mad_cleanup(ibdev); + ib_unregister_device(&ibdev->ib_dev); + for (p = 1; p <= ibdev->num_ports; ++p) mlx4_CLOSE_PORT(dev, p); - mlx4_ib_mad_cleanup(ibdev); - ib_unregister_device(&ibdev->ib_dev); iounmap(ibdev->uar_map); mlx4_uar_free(dev, &ibdev->priv_uar); mlx4_pd_free(dev, ibdev->priv_pdn); -- 1.6.1.3 From michael.heinz at qlogic.com Tue Mar 17 09:07:24 2009 From: michael.heinz at qlogic.com (Mike Heinz) Date: Tue, 17 Mar 2009 11:07:24 -0500 Subject: [ofa-general] Weird problem with calling libibverbs from another .SO Message-ID: <4C2744E8AD2982428C5BFE523DF8CDCB43E6D5A58C@MNEXMB1.qlogic.org> I'm working on some sample programs and I've run into a weird problem with libibverbs. If I have a simple program linked with libibverbs, things worked smoothly, but if I more those functions into a shared library which then calls libibverbs, I core dump, with what appears to be a corrupted call stack. Are there special flags I need to use when creating an SO that works with libibverbs? Here's what I mean: hca = open_device(argv[optind]); // value of hca is 0x5054a0. if (!hca) { fprintf(stderr, "Could not open HCA.\n"); return -1; } { struct ibv_device_attr attr; ibv_query_device(hca,&attr); // this call completes successfully. fprintf(stderr,"MHEINZ: Firmware = %s\n", attr.fw_ver); } err = my_shared_func(hca); ... int my_shared_func(struct ibv_context *hca) { struct ibv_device_attr attr; hca_context->ops.query_device(hca_context, &attr); // This works! ibv_query_device(hca_context,&attr); // core dump here. fprintf(stderr,"MHEINZ: Firmware = %s\n", attr.fw_ver); return 0; } First, note that calling the device query directly, by using the function pointers in hca_context, works - so I know that the value of hca_context is correct. On the other hand, at the point of the core dump, gdb reports that inside ibv_query_device, the value of hca_context has changed from 0x5044a0 to 0x2b46392fb310! -- Michael Heinz Principal Engineer, Qlogic Corporation King of Prussia, Pennsylvania -------------- next part -------------- An HTML attachment was scrubbed... URL: From dwelp at mail.uni-paderborn.de Tue Mar 17 09:06:16 2009 From: dwelp at mail.uni-paderborn.de (dwelp at mail.uni-paderborn.de) Date: Tue, 17 Mar 2009 17:06:16 +0100 Subject: [ofa-general] XenIB: Error inserting ib_core: Invalid module format Message-ID: <20090317170616.nge56lgaomcg8s00@webmail.uni-paderborn.de> Hi, for some reason I cannot load the ib_core module, any help or comment would be greatly appreciated! The system runs an UbuntuServer 6.10 (2.6.17-10) paravirtualized with Xen-3.0.0 (XenIB) from OSU [as described in Panda: High Performance VMM-Bypass I/O in Virtual Machines (Usenix 06)]. The Xen is from the http://xenbits.xensource.com/ext/xen-smartio.hg repository. It is configured with Infiniband support and successfully builded the enclosed modules. root at ic13:~# uname -a Linux ic13 2.6.16-rc3-xen0 #1 Tue Jan 20 16:43:16 CET 2009 x86_64 GNU/Linux I downloaded and untared: http://www.openfabrics.org/~vlad/builds/ofa_1_2_user/ofa_1_2_user-20070623-0200.tgz http://www.openfabrics.org/~vlad/builds/ofa_1_2_kernel/ofa_1_2_kernel-20070822-0100.tgz userspace went through perfectly with the following flow: root at ic13:~# ./configure --with-libibverbs --with-mthca --with-libibcm --with-librdmacm --with-management-libs --with-openib-diags --with-opensm --with-ipoibtools root at ic13:~# make & make install constructing the modules runs without any errors: root at ic13:~# ./configure --with-core-mod --with-ipoib-mod --with-ipoib-cm --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mthca-mod root at ic13:~# make & make install & depmod the config-file: root at ic13:~/ofed/ofa_1_2_kernel-20070822-0100# vim configure.mk.kernel # Current working directory CWD=/home/dwelp/ofed/ofa_1_2_kernel-20070822-0100 # Kernel level KVERSION=2.6.16-rc3-xen0 ARCH=x86_64 EXTRAVERSION=-rc3-xen0 MODULES_DIR=/lib/modules/2.6.16-rc3-xen0/updates KSRC=/lib/modules/2.6.16-rc3-xen0/build BACKPORT_INCLUDES=-I/home/dwelp/ofed/ofa_1_2_kernel-20070822-0100/kernel_addons/backport/2.6.16/include/ AUTOCONF_H=/home/dwelp/ofed/ofa_1_2_kernel-20070822-0100/include/linux/autoconf.h WITH_MEMTRACK=no WITH_MAKE_PARAMS= CONFIG_INFINIBAND=m CONFIG_INFINIBAND_IPOIB=m CONFIG_INFINIBAND_IPOIB_CM=y CONFIG_INFINIBAND_SDP= CONFIG_INFINIBAND_SRP= CONFIG_INFINIBAND_USER_MAD=m CONFIG_INFINIBAND_USER_ACCESS=m CONFIG_INFINIBAND_ADDR_TRANS=y CONFIG_INFINIBAND_MTHCA=m CONFIG_INFINIBAND_IPOIB_DEBUG=y CONFIG_INFINIBAND_ISER= CONFIG_SCSI_ISCSI_ATTRS= CONFIG_ISCSI_TCP= CONFIG_INFINIBAND_EHCA= CONFIG_INFINIBAND_EHCA_SCALING= CONFIG_RDS= CONFIG_RDS_IB= CONFIG_RDS_TCP= CONFIG_RDS_DEBUG= CONFIG_INFINIBAND_MADEYE= CONFIG_INFINIBAND_VNIC= CONFIG_INFINIBAND_CXGB3= CONFIG_CHELSIO_T3= CONFIG_INFINIBAND_IPOIB_DEBUG_DATA= CONFIG_INFINIBAND_SDP_SEND_ZCOPY= CONFIG_INFINIBAND_SDP_RECV_ZCOPY= CONFIG_INFINIBAND_SDP_DEBUG= CONFIG_INFINIBAND_SDP_DEBUG_DATA= CONFIG_INFINIBAND_IPATH= CONFIG_INFINIBAND_MTHCA_DEBUG=y CONFIG_INFINIBAND_VNIC_DEBUG= CONFIG_INFINIBAND_VNIC_STATS= CONFIG_INFINIBAND_CXGB3_DEBUG= but when I try to load them it results in this: root at ic13:~# modprobe ib_mthca WARNING: Error inserting ib_core (/lib/modules/2.6.16-rc3-xen0/updates/kernel/drivers/infiniband/core/ib_core.ko): Invalid module format WARNING: Error inserting ib_mad (/lib/modules/2.6.16-rc3-xen0/updates/kernel/drivers/infiniband/core/ib_mad.ko): Unknown symbol in module, or unknown parameter (see dmesg) FATAL: Error inserting ib_mthca (/lib/modules/2.6.16-rc3-xen0/updates/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko): Unknown symbol in module, or unknown parameter (see dmesg) the dmesg shows: ib_core: exports duplicate symbol ib_unpack (owned by kernel) ib_mad: Unknown symbol rdma_node_get_transport ib_mad: Unknown symbol ib_get_cached_lmc ib_mthca: Unknown symbol mult_to_ib_rate ib_mthca: Unknown symbol ib_modify_qp_is_ok ib_mthca: Unknown symbol ib_rate_to_mult ib_mthca: Unknown symbol dma_get_cache_alignment root at ic13:~# modinfo ib_core filename: /lib/modules/2.6.16-rc3-xen0/updates/kernel/drivers/infiniband/core/ib_core.ko license: Dual BSD/GPL description: core kernel InfiniBand API author: Roland Dreier depends: vermagic: 2.6.16-rc3-xen0 gcc-3.4 Any ideas? Thanks in advance! Daniel Welp From sashak at voltaire.com Tue Mar 17 12:08:17 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Mar 2009 21:08:17 +0200 Subject: [ofa-general] Re: [PATCHv2 1/2] opensm: Add common ib_gid_is_notzero routine In-Reply-To: <20090312201918.GA25024@comcast.net> References: <20090312201918.GA25024@comcast.net> Message-ID: <20090317190817.GB604@sashak.voltaire.com> On 15:19 Thu 12 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied (with changes described below). Thanks. > diff --git a/opensm/include/opensm/osm_helper.h b/opensm/include/opensm/osm_helper.h > index 9222853..cbb6408 100644 > --- a/opensm/include/opensm/osm_helper.h > +++ b/opensm/include/opensm/osm_helper.h > @@ -2,6 +2,7 @@ > * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. > * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. > + * Copyright (c) 2009 HNR Consulting. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -168,6 +169,40 @@ const char *ib_get_trap_str(uint16_t trap_num); > * > *********/ > > +OSM_EXPORT const ib_gid_t ib_zero_gid; > + > +/****f* IBA Base: Types/ib_gid_is_notzero > +* NAME > +* ib_gid_is_notzero > +* > +* DESCRIPTION > +* Returns a boolean indicating whether or not the GID is zero. > +* > +* SYNOPSIS > +*/ > +static inline boolean_t OSM_API OSM_EXPORT and OSM_API were never used in osm_helper.h, I removed it. > +ib_gid_is_notzero(IN const ib_gid_t * const p_gid) Second 'const' means that this function cannot change p_gid pointer value. It is almost useless in C where parameters are passed by value anyway. > +{ > + if (memcmp(p_gid, &ib_zero_gid, sizeof(*p_gid))) > + return TRUE; > + return FALSE; Did this as 'return !memcmp(....)' to simplify the flow. Sasha From Ted.Kim at Sun.COM Tue Mar 17 12:02:48 2009 From: Ted.Kim at Sun.COM (Ted H. Kim) Date: Tue, 17 Mar 2009 12:02:48 -0700 Subject: [ofa-general] 4K MTU with OpenSM? Message-ID: <49BFF3D8.4040402@sun.com> Folks, Is there way to config 4K IB MTU with OpenSM? What's the option of file setting for that? I saw something about MC MTU in the manpage, but not a general setting for all the links. Admitedly, I may not have been looking at the most recent version. Thanks, -ted -- Ted H. Kim Sun Microsystems, Inc. ted.kim at sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From hal.rosenstock at gmail.com Tue Mar 17 12:42:54 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 17 Mar 2009 15:42:54 -0400 Subject: ***SPAM*** Re: [ofa-general] 4K MTU with OpenSM? In-Reply-To: <49BFF3D8.4040402@sun.com> References: <49BFF3D8.4040402@sun.com> Message-ID: Ted, On Tue, Mar 17, 2009 at 3:02 PM, Ted H. Kim wrote: > Folks, > > Is there way to config 4K IB MTU with OpenSM? > What's the option of file setting for that? See the partition configuration section in the man page or opensm/doc/partition-config.txt. It's something like: Default=0x7fff,ipoib,mtu=5:ALL=full; assuming you want this for IPoIB on the default partition. > I saw something about MC MTU in the manpage, but > not a general setting for all the links. Admitedly, > I may not have been looking at the most recent > version. This syntax has been supported for quite a while now so I would expect it to be supported by most OpenSM versions being used. -- Hal > Thanks, > -ted > > -- > Ted H. Kim > Sun Microsystems, Inc.                  ted.kim at sun.com > 222 North Sepulveda Blvd., 10th Floor   (310) 341-1116 > El Segundo, CA  90245                   (310) 341-1120 FAX > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From sashak at voltaire.com Tue Mar 17 12:54:59 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Mar 2009 21:54:59 +0200 Subject: [ofa-general] Re: [PATCHv2 1/2] opensm: Add common ib_gid_is_notzero routine In-Reply-To: <20090317190817.GB604@sashak.voltaire.com> References: <20090312201918.GA25024@comcast.net> <20090317190817.GB604@sashak.voltaire.com> Message-ID: <20090317195459.GC604@sashak.voltaire.com> On 21:08 Tue 17 Mar , Sasha Khapyorsky wrote: > > +{ > > + if (memcmp(p_gid, &ib_zero_gid, sizeof(*p_gid))) > > + return TRUE; > > + return FALSE; > > Did this as 'return !memcmp(....)' to simplify the flow. Assuming 'return memcmp()' of course :) Sasha From sashak at voltaire.com Tue Mar 17 13:15:17 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 17 Mar 2009 22:15:17 +0200 Subject: [ofa-general] Re: [PATCHv2 2/2] opensm: Utilize ib_gid_is_notzero routine In-Reply-To: <20090312202008.GB25024@comcast.net> References: <20090312202008.GB25024@comcast.net> Message-ID: <20090317201457.GD604@sashak.voltaire.com> On 15:20 Thu 12 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From hal.rosenstock at gmail.com Tue Mar 17 14:40:52 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 17 Mar 2009 17:40:52 -0400 Subject: ***SPAM*** Re: [ofa-general] Need help with Infiniband problem In-Reply-To: <49BFBA94.5000709@uwyo.edu> References: <49BFBA94.5000709@uwyo.edu> Message-ID: 2009/3/17 jeffrey Lang : > First let me say, I hope this is the right list for this email, if not > please forgive me. > > I have a small 16 node compute cluster.    The university where I work at > recently opened a new Datacenter.  My cluster was moved from the old > Datacenter.   Before the move the inifiniband was working properly, after > the move the ipoib has stopped working. > > The cluster runs Centos 4 with all the latest updates and the Centos > distributed OFED code.   My plan was to update the OFED code once things had > restablized. > > For the move, I shutdown the cluster, removed the inifiniband cables and the > cluster was moved.   I then reinstalled the infiniband cables (not in the > same order before the move) and brought every thing back up. > > When i brought the cluster back up the ipoib would not work.  The only > message in the log file is "Mar 15 04:04:32 h2o01 kernel: ib0: multicast > join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22". I think that there may be a rate issue in terms of this node relative to the IPoIB broadcast group which by default is 10 Gbps (4x SDR). What does this node's portinfo show (smpquery portinfo -D 0) in terms of link width and speed ? -- Hal > The master node can see all the systems: > > [root at h2o01 log]# ibnodes > Ca    : 0x00066a0098007e99 ports 1 "h2o17 HCA-1" > Ca    : 0x00066a0098007e9b ports 1 "h2o18 HCA-1" > Ca    : 0x00066a0098007e97 ports 1 "h2o16 HCA-1" > Ca    : 0x00066a0098007e8c ports 1 "h2o15 HCA-1" > Ca    : 0x00066a0098007e94 ports 1 "h2o14 HCA-1" > Ca    : 0x00066a0098007e93 ports 1 "h2o13 HCA-1" > Ca    : 0x00066a0098007e8e ports 1 "h2o12 HCA-1" > Ca    : 0x00066a0098007e90 ports 1 "h2o11 HCA-1" > Ca    : 0x00066a0098007e98 ports 1 "h2o10 HCA-1" > Ca    : 0x00066a0098007e95 ports 1 "h2o09 HCA-1" > Ca    : 0x00066a0098007e8f ports 1 "h2o08 HCA-1" > Ca    : 0x00066a0098007e92 ports 1 "h2o07 HCA-1" > Ca    : 0x00066a0098007e8d ports 1 "h2o06 HCA-1" > Ca    : 0x00066a0098007e91 ports 1 "h2o05 HCA-1" > Ca    : 0x00066a0098007e96 ports 1 "h2ocfs HCA-1" > Ca    : 0x00066a0098007e9c ports 1 "h2o01 HCA-1" > Switch    : 0x00066a00d8000593 ports 24 "SilverStorm 9024 > GUID=0x00066a00d8000593" enhanced port 0 lid 1 lmc 0 > > I've reset the sm on the switch, but nothing seems to work. > > Any ideas of where to look for whats causing the problem? > > jeff > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From jrlang at uwyo.edu Tue Mar 17 15:04:11 2009 From: jrlang at uwyo.edu (jeffrey Lang) Date: Tue, 17 Mar 2009 16:04:11 -0600 Subject: [ofa-general] Need help with Infiniband problem In-Reply-To: References: <49BFBA94.5000709@uwyo.edu> Message-ID: <49C01E5B.5070703@uwyo.edu> Here's the output smpquery portinfo -D 0 as requested below: [root at h2o01 ~]# smpquery portinfo -D 0 # Port info: DR path 0 port 0 Mkey:............................0x0000000000000000 GidPrefix:.......................0xfe80000000000000 Lid:.............................0x0003 SMLid:...........................0x0001 CapMask:.........................0x2510a68 IsTrapSupported IsAutomaticMigrationSupported IsSLMappingSupported IsLedInfoSupported IsSystemImageGUIDsupported IsCommunicatonManagementSupported IsVendorClassSupported IsCapabilityMaskNoticeSupported IsClientRegistrationSupported DiagCode:........................0x0000 MkeyLeasePeriod:.................0 LocalPort:.......................1 LinkWidthEnabled:................1X or 4X LinkWidthSupported:..............1X or 4X LinkWidthActive:.................4X LinkSpeedSupported:..............2.5 Gbps LinkState:.......................Active PhysLinkState:...................LinkUp LinkDownDefState:................Polling ProtectBits:.....................0 LMC:.............................0 LinkSpeedActive:.................2.5 Gbps LinkSpeedEnabled:................2.5 Gbps NeighborMTU:.....................2048 SMSL:............................0 VLCap:...........................VL0-3 InitType:........................0x00 VLHighLimit:.....................0 VLArbHighCap:....................8 VLArbLowCap:.....................8 InitReply:.......................0x00 MtuCap:..........................2048 VLStallCount:....................7 HoqLife:.........................0 OperVLs:.........................VL0-3 PartEnforceInb:..................0 PartEnforceOutb:.................0 FilterRawInb:....................0 FilterRawOutb:...................0 MkeyViolations:..................0 PkeyViolations:..................0 QkeyViolations:..................0 GuidCap:.........................32 ClientReregister:................0 SubnetTimeout:...................17 RespTimeVal:.....................16 LocalPhysErr:....................15 OverrunErr:......................15 MaxCreditHint:...................0 RoundTrip:.......................0 I did some checking, and It's not just this node having problems, all nodes seem to be having this same problem. jeff Hal Rosenstock wrote: > 2009/3/17 jeffrey Lang : > >> First let me say, I hope this is the right list for this email, if not >> please forgive me. >> >> I have a small 16 node compute cluster. The university where I work at >> recently opened a new Datacenter. My cluster was moved from the old >> Datacenter. Before the move the inifiniband was working properly, after >> the move the ipoib has stopped working. >> >> The cluster runs Centos 4 with all the latest updates and the Centos >> distributed OFED code. My plan was to update the OFED code once things had >> restablized. >> >> For the move, I shutdown the cluster, removed the inifiniband cables and the >> cluster was moved. I then reinstalled the infiniband cables (not in the >> same order before the move) and brought every thing back up. >> >> When i brought the cluster back up the ipoib would not work. The only >> message in the log file is "Mar 15 04:04:32 h2o01 kernel: ib0: multicast >> join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22". >> > > I think that there may be a rate issue in terms of this node relative > to the IPoIB broadcast group which by default is 10 Gbps (4x SDR). > What does this node's portinfo show (smpquery portinfo -D 0) in terms > of link width and speed ? > > -- Hal > > >> The master node can see all the systems: >> >> [root at h2o01 log]# ibnodes >> Ca : 0x00066a0098007e99 ports 1 "h2o17 HCA-1" >> Ca : 0x00066a0098007e9b ports 1 "h2o18 HCA-1" >> Ca : 0x00066a0098007e97 ports 1 "h2o16 HCA-1" >> Ca : 0x00066a0098007e8c ports 1 "h2o15 HCA-1" >> Ca : 0x00066a0098007e94 ports 1 "h2o14 HCA-1" >> Ca : 0x00066a0098007e93 ports 1 "h2o13 HCA-1" >> Ca : 0x00066a0098007e8e ports 1 "h2o12 HCA-1" >> Ca : 0x00066a0098007e90 ports 1 "h2o11 HCA-1" >> Ca : 0x00066a0098007e98 ports 1 "h2o10 HCA-1" >> Ca : 0x00066a0098007e95 ports 1 "h2o09 HCA-1" >> Ca : 0x00066a0098007e8f ports 1 "h2o08 HCA-1" >> Ca : 0x00066a0098007e92 ports 1 "h2o07 HCA-1" >> Ca : 0x00066a0098007e8d ports 1 "h2o06 HCA-1" >> Ca : 0x00066a0098007e91 ports 1 "h2o05 HCA-1" >> Ca : 0x00066a0098007e96 ports 1 "h2ocfs HCA-1" >> Ca : 0x00066a0098007e9c ports 1 "h2o01 HCA-1" >> Switch : 0x00066a00d8000593 ports 24 "SilverStorm 9024 >> GUID=0x00066a00d8000593" enhanced port 0 lid 1 lmc 0 >> >> I've reset the sm on the switch, but nothing seems to work. >> >> Any ideas of where to look for whats causing the problem? >> >> jeff >> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: jrlang.vcf Type: text/x-vcard Size: 311 bytes Desc: not available URL: From hal.rosenstock at gmail.com Tue Mar 17 15:22:45 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 17 Mar 2009 18:22:45 -0400 Subject: ***SPAM*** Re: [ofa-general] Need help with Infiniband problem In-Reply-To: <49C01E5B.5070703@uwyo.edu> References: <49BFBA94.5000709@uwyo.edu> <49C01E5B.5070703@uwyo.edu> Message-ID: On Tue, Mar 17, 2009 at 6:04 PM, jeffrey Lang wrote: > Here's the output smpquery portinfo -D 0 as requested below: > > [root at h2o01 ~]# smpquery portinfo -D 0 > # Port info: DR path 0 port 0 > Mkey:............................0x0000000000000000 > GidPrefix:.......................0xfe80000000000000 > Lid:.............................0x0003 > SMLid:...........................0x0001 > CapMask:.........................0x2510a68 >                 IsTrapSupported >                 IsAutomaticMigrationSupported >                 IsSLMappingSupported >                 IsLedInfoSupported >                 IsSystemImageGUIDsupported >                 IsCommunicatonManagementSupported >                 IsVendorClassSupported >                 IsCapabilityMaskNoticeSupported >                 IsClientRegistrationSupported > DiagCode:........................0x0000 > MkeyLeasePeriod:.................0 > LocalPort:.......................1 > LinkWidthEnabled:................1X or 4X > LinkWidthSupported:..............1X or 4X > LinkWidthActive:.................4X > LinkSpeedSupported:..............2.5 Gbps > LinkState:.......................Active > PhysLinkState:...................LinkUp > LinkDownDefState:................Polling > ProtectBits:.....................0 > LMC:.............................0 > LinkSpeedActive:.................2.5 Gbps > LinkSpeedEnabled:................2.5 Gbps > NeighborMTU:.....................2048 > SMSL:............................0 > VLCap:...........................VL0-3 > InitType:........................0x00 > VLHighLimit:.....................0 > VLArbHighCap:....................8 > VLArbLowCap:.....................8 > InitReply:.......................0x00 > MtuCap:..........................2048 > VLStallCount:....................7 > HoqLife:.........................0 > OperVLs:.........................VL0-3 > PartEnforceInb:..................0 > PartEnforceOutb:.................0 > FilterRawInb:....................0 > FilterRawOutb:...................0 > MkeyViolations:..................0 > PkeyViolations:..................0 > QkeyViolations:..................0 > GuidCap:.........................32 > ClientReregister:................0 > SubnetTimeout:...................17 > RespTimeVal:.....................16 > LocalPhysErr:....................15 > OverrunErr:......................15 > MaxCreditHint:...................0 > RoundTrip:.......................0 Looks fine. > I did some checking, and It's not just this node having problems, all nodes > seem to be having this same problem. Would you also run ibchecknet ? What error messages are on the SM side ? -- Hal > jeff > > > Hal Rosenstock wrote: > > 2009/3/17 jeffrey Lang : > > > First let me say, I hope this is the right list for this email, if not > please forgive me. > I have a small 16 node compute cluster.    The university where I work at > recently opened a new Datacenter.  My cluster was moved from the old > Datacenter.   Before the move the inifiniband was working properly, after > the move the ipoib has stopped working. > The cluster runs Centos 4 with all the latest updates and the Centos > distributed OFED code.   My plan was to update the OFED code once things had > restablized. > For the move, I shutdown the cluster, removed the inifiniband cables and the > cluster was moved.   I then reinstalled the infiniband cables (not in the > same order before the move) and brought every thing back up. > When i brought the cluster back up the ipoib would not work.  The only > message in the log file is "Mar 15 04:04:32 h2o01 kernel: ib0: multicast > join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22". > > > I think that there may be a rate issue in terms of this node relative > to the IPoIB broadcast group which by default is 10 Gbps (4x SDR). > What does this node's portinfo show (smpquery portinfo -D 0) in terms > of link width and speed ? > -- Hal > > > The master node can see all the systems: > [root at h2o01 log]# ibnodes > Ca    : 0x00066a0098007e99 ports 1 "h2o17 HCA-1" > Ca    : 0x00066a0098007e9b ports 1 "h2o18 HCA-1" > Ca    : 0x00066a0098007e97 ports 1 "h2o16 HCA-1" > Ca    : 0x00066a0098007e8c ports 1 "h2o15 HCA-1" > Ca    : 0x00066a0098007e94 ports 1 "h2o14 HCA-1" > Ca    : 0x00066a0098007e93 ports 1 "h2o13 HCA-1" > Ca    : 0x00066a0098007e8e ports 1 "h2o12 HCA-1" > Ca    : 0x00066a0098007e90 ports 1 "h2o11 HCA-1" > Ca    : 0x00066a0098007e98 ports 1 "h2o10 HCA-1" > Ca    : 0x00066a0098007e95 ports 1 "h2o09 HCA-1" > Ca    : 0x00066a0098007e8f ports 1 "h2o08 HCA-1" > Ca    : 0x00066a0098007e92 ports 1 "h2o07 HCA-1" > Ca    : 0x00066a0098007e8d ports 1 "h2o06 HCA-1" > Ca    : 0x00066a0098007e91 ports 1 "h2o05 HCA-1" > Ca    : 0x00066a0098007e96 ports 1 "h2ocfs HCA-1" > Ca    : 0x00066a0098007e9c ports 1 "h2o01 HCA-1" > Switch    : 0x00066a00d8000593 ports 24 "SilverStorm 9024 > GUID=0x00066a00d8000593" enhanced port 0 lid 1 lmc 0 > I've reset the sm on the switch, but nothing seems to work. > Any ideas of where to look for whats causing the problem? > jeff > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From hal.rosenstock at gmail.com Tue Mar 17 16:05:57 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 17 Mar 2009 19:05:57 -0400 Subject: ***SPAM*** Re: ***SPAM*** Re: [ofa-general] Need help with Infiniband problem In-Reply-To: References: <49BFBA94.5000709@uwyo.edu> <49C01E5B.5070703@uwyo.edu> Message-ID: On Tue, Mar 17, 2009 at 6:22 PM, Hal Rosenstock wrote: > Would you also run ibchecknet ? > > What error messages are on the SM side ? What does saquery -g show ? You did say this was a switch based SM, right ? -- Hal From dave.olson at qlogic.com Tue Mar 17 18:17:50 2009 From: dave.olson at qlogic.com (Dave Olson) Date: Tue, 17 Mar 2009 18:17:50 -0700 Subject: [ofa-general] 4K MTU with OpenSM? In-Reply-To: References: <49BFF3D8.4040402@sun.com> Message-ID: On Tue, 17 Mar 2009, Hal Rosenstock wrote: | On Tue, Mar 17, 2009 at 3:02 PM, Ted H. Kim wrote: | > Is there way to config 4K IB MTU with OpenSM? | > What's the option of file setting for that? | | See the partition configuration section in the man page or | opensm/doc/partition-config.txt. It's something like: | | Default=0x7fff,ipoib,mtu=5:ALL=full; | | assuming you want this for IPoIB on the default partition. Or even if you don't care about it for ipoib, just for other things (just to clarify, as the ipoib part of it confused me at first). | This syntax has been supported for quite a while now so I would expect | it to be supported by most OpenSM versions being used. It's worked fine for our testing of the qlogic infinipath HCAs for ofed 1.3 and ofed 1.4 versions of opensm. Dave Olson dave.olson at qlogic.com From ogerlitz at voltaire.com Wed Mar 18 01:18:36 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Wed, 18 Mar 2009 10:18:36 +0200 (IST) Subject: [ofa-general] possible bugs in IB code usage of min() and max() Message-ID: Roland, I came across these sparse warnings, are they false-negative or bugs that should be fixed? drivers/infiniband/hw/mlx4/qp.c:421:22: warning: symbol '_min1' shadows an earlier one drivers/infiniband/hw/mlx4/qp.c:421:22: warning: symbol '_min1' shadows an earlier one drivers/infiniband/hw/mlx4/qp.c:421:22: warning: symbol '_min2' shadows an earlier one drivers/infiniband/ulp/ipoib/ipoib_main.c:1399:21: warning: symbol '_max1' shadows an earlier one drivers/infiniband/ulp/ipoib/ipoib_main.c:1399:21: warning: symbol '_max1' shadows an earlier one drivers/infiniband/ulp/ipoib/ipoib_main.c:1399:21: warning: symbol '_max2' shadows an earlier one Or. From sashak at voltaire.com Wed Mar 18 02:44:12 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 18 Mar 2009 11:44:12 +0200 Subject: [ofa-general] 4K MTU with OpenSM? In-Reply-To: References: <49BFF3D8.4040402@sun.com> Message-ID: <20090318094412.GF604@sashak.voltaire.com> On 18:17 Tue 17 Mar , Dave Olson wrote: > On Tue, 17 Mar 2009, Hal Rosenstock wrote: > | On Tue, Mar 17, 2009 at 3:02 PM, Ted H. Kim wrote: > | > Is there way to config 4K IB MTU with OpenSM? > | > What's the option of file setting for that? > | > | See the partition configuration section in the man page or > | opensm/doc/partition-config.txt. It's something like: > | > | Default=0x7fff,ipoib,mtu=5:ALL=full; This will use this MTU value for IPoIB multicast group. As far as I understood Ted's question was "how to setup port's MTU using OpenSM". Assuming so I don't think that OpenSM has such functionality now (although it should not be very hard to add). Sasha From dorfman.eli at gmail.com Wed Mar 18 03:00:43 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Wed, 18 Mar 2009 12:00:43 +0200 Subject: [ofa-general] ***SPAM*** [PATCH] add c99 definitions within the ib_mad_f structure In-Reply-To: <20090309113530.8b056000.weiny2@llnl.gov> References: <49B4CFC6.7020800@gmail.com> <20090309154000.GL15398@sashak.voltaire.com> <8436BA8017D247AC9287E3F0DB4458DB@amr.corp.intel.com> <20090309113233.b382dc70.weiny2@llnl.gov> <20090309113530.8b056000.weiny2@llnl.gov> Message-ID: <49C0C64B.6050606@gmail.com> add c99 definitions to ib_mad_f Signed-off-by: Eli Dorfman --- libibmad/src/fields.c | 525 ++++++++++++++++++++++++------------------------- 1 files changed, 262 insertions(+), 263 deletions(-) diff --git a/libibmad/src/fields.c b/libibmad/src/fields.c index 19c8fc1..c142f39 100644 --- a/libibmad/src/fields.c +++ b/libibmad/src/fields.c @@ -54,10 +54,10 @@ #define BE_TO_BITSOFFS(o, w) (((o) & ~31) | ((32 - ((o) & 31) - (w)))) static const ib_field_t ib_mad_f[] = { - {0, 0}, /* IB_NO_FIELD - reserved as invalid */ + [ IB_NO_FIELD ] {0, 0}, /* IB_NO_FIELD - reserved as invalid */ - {0, 64, "GidPrefix", mad_dump_rhex}, - {64, 64, "GidGuid", mad_dump_rhex}, + [ IB_GID_PREFIX_F ] {0, 64, "GidPrefix", mad_dump_rhex}, + [ IB_GID_GUID_F ] {64, 64, "GidGuid", mad_dump_rhex}, /* * MAD: common MAD fields (IB spec 13.4.2) @@ -67,379 +67,378 @@ static const ib_field_t ib_mad_f[] = { */ /* first MAD word (0-3 bytes) */ - {BE_OFFS(0, 7), "MadMethod", mad_dump_hex}, /* TODO: add dumper */ - {BE_OFFS(7, 1), "MadIsResponse", mad_dump_uint}, /* TODO: add dumper */ - {BE_OFFS(8, 8), "MadClassVersion", mad_dump_uint}, - {BE_OFFS(16, 8), "MadMgmtClass", mad_dump_uint}, /* TODO: add dumper */ - {BE_OFFS(24, 8), "MadBaseVersion", mad_dump_uint}, + [ IB_MAD_METHOD_F ] {BE_OFFS(0, 7), "MadMethod", mad_dump_hex}, /* TODO: add dumper */ + [ IB_MAD_RESPONSE_F ] {BE_OFFS(7, 1), "MadIsResponse", mad_dump_uint}, /* TODO: add dumper */ + [ IB_MAD_CLASSVER_F ] {BE_OFFS(8, 8), "MadClassVersion", mad_dump_uint}, + [ IB_MAD_MGMTCLASS_F ] {BE_OFFS(16, 8), "MadMgmtClass", mad_dump_uint}, /* TODO: add dumper */ + [ IB_MAD_BASEVER_F ] {BE_OFFS(24, 8), "MadBaseVersion", mad_dump_uint}, /* second MAD word (4-7 bytes) */ - {BE_OFFS(48, 16), "MadStatus", mad_dump_hex}, /* TODO: add dumper */ + [ IB_MAD_STATUS_F ] {BE_OFFS(48, 16), "MadStatus", mad_dump_hex}, /* TODO: add dumper */ /* DR SMP only */ - {BE_OFFS(32, 8), "DrSmpHopCnt", mad_dump_uint}, - {BE_OFFS(40, 8), "DrSmpHopPtr", mad_dump_uint}, - {BE_OFFS(48, 15), "DrSmpStatus", mad_dump_hex}, /* TODO: add dumper */ - {BE_OFFS(63, 1), "DrSmpDirection", mad_dump_uint}, /* TODO: add dumper */ + [ IB_DRSMP_HOPCNT_F ] {BE_OFFS(32, 8), "DrSmpHopCnt", mad_dump_uint}, + [ IB_DRSMP_HOPPTR_F ] {BE_OFFS(40, 8), "DrSmpHopPtr", mad_dump_uint}, + [ IB_DRSMP_STATUS_F ] {BE_OFFS(48, 15), "DrSmpStatus", mad_dump_hex}, /* TODO: add dumper */ + [ IB_DRSMP_DIRECTION_F ] {BE_OFFS(63, 1), "DrSmpDirection", mad_dump_uint}, /* TODO: add dumper */ /* words 3,4,5,6 (8-23 bytes) */ - {64, 64, "MadTRID", mad_dump_hex}, - {BE_OFFS(144, 16), "MadAttr", mad_dump_hex}, /* TODO: add dumper */ - {160, 32, "MadModifier", mad_dump_hex}, /* TODO: add dumper */ + [ IB_MAD_TRID_F ] {64, 64, "MadTRID", mad_dump_hex}, + [ IB_MAD_ATTRID_F ] {BE_OFFS(144, 16), "MadAttr", mad_dump_hex}, /* TODO: add dumper */ + [ IB_MAD_ATTRMOD_F ] {160, 32, "MadModifier", mad_dump_hex}, /* TODO: add dumper */ /* word 7,8 (24-31 bytes) */ - {192, 64, "MadMkey", mad_dump_hex}, + [ IB_MAD_MKEY_F ] {192, 64, "MadMkey", mad_dump_hex}, /* word 9 (32-37 bytes) */ - {BE_OFFS(256, 16), "DrSmpDLID", mad_dump_hex}, - {BE_OFFS(272, 16), "DrSmpSLID", mad_dump_hex}, + [ IB_DRSMP_DRSLID_F ] {BE_OFFS(256, 16), "DrSmpDLID", mad_dump_hex}, + [ IB_DRSMP_DRDLID_F ] {BE_OFFS(272, 16), "DrSmpSLID", mad_dump_hex}, /* word 10,11 (36-43 bytes) */ - {288, 64, "SaSMkey", mad_dump_hex}, + [ IB_SA_MKEY_F ] {288, 64, "SaSMkey", mad_dump_hex}, /* word 12 (44-47 bytes) */ - {BE_OFFS(46 * 8, 16), "SaAttrOffs", mad_dump_uint}, + [ IB_SA_ATTROFFS_F ] {BE_OFFS(46 * 8, 16), "SaAttrOffs", mad_dump_uint}, /* word 13,14 (48-55 bytes) */ - {48 * 8, 64, "SaCompMask", mad_dump_hex}, + [ IB_SA_COMPMASK_F ] {48 * 8, 64, "SaCompMask", mad_dump_hex}, /* word 13,14 (56-255 bytes) */ - {56 * 8, (256 - 56) * 8, "SaData", mad_dump_hex}, + [ IB_SA_DATA_F ] {56 * 8, (256 - 56) * 8, "SaData", mad_dump_hex}, /* bytes 64 - 127 */ - {0, 0}, /* IB_SM_DATA_F - reserved as invalid */ + [ IB_SM_DATA_F ] {0, 0}, /* IB_SM_DATA_F - reserved as invalid */ /* bytes 64 - 256 */ - {64 * 8, (256 - 64) * 8, "GsData", mad_dump_hex}, + [ IB_GS_DATA_F ] {64 * 8, (256 - 64) * 8, "GsData", mad_dump_hex}, /* bytes 128 - 191 */ - {1024, 512, "DrSmpPath", mad_dump_hex}, + [ IB_DRSMP_PATH_F ] {1024, 512, "DrSmpPath", mad_dump_hex}, /* bytes 192 - 255 */ - {1536, 512, "DrSmpRetPath", mad_dump_hex}, + [ IB_DRSMP_RPATH_F ] {1536, 512, "DrSmpRetPath", mad_dump_hex}, /* * PortInfo fields */ - {0, 64, "Mkey", mad_dump_hex}, - {64, 64, "GidPrefix", mad_dump_hex}, - {BITSOFFS(128, 16), "Lid", mad_dump_uint}, - {BITSOFFS(144, 16), "SMLid", mad_dump_uint}, - {160, 32, "CapMask", mad_dump_portcapmask}, - {BITSOFFS(192, 16), "DiagCode", mad_dump_hex}, - {BITSOFFS(208, 16), "MkeyLeasePeriod", mad_dump_uint}, - {BITSOFFS(224, 8), "LocalPort", mad_dump_uint}, - {BITSOFFS(232, 8), "LinkWidthEnabled", mad_dump_linkwidthen}, - {BITSOFFS(240, 8), "LinkWidthSupported", mad_dump_linkwidthsup}, - {BITSOFFS(248, 8), "LinkWidthActive", mad_dump_linkwidth}, - {BITSOFFS(256, 4), "LinkSpeedSupported", mad_dump_linkspeedsup}, - {BITSOFFS(260, 4), "LinkState", mad_dump_portstate}, - {BITSOFFS(264, 4), "PhysLinkState", mad_dump_physportstate}, - {BITSOFFS(268, 4), "LinkDownDefState", mad_dump_linkdowndefstate}, - {BITSOFFS(272, 2), "ProtectBits", mad_dump_uint}, - {BITSOFFS(277, 3), "LMC", mad_dump_uint}, - {BITSOFFS(280, 4), "LinkSpeedActive", mad_dump_linkspeed}, - {BITSOFFS(284, 4), "LinkSpeedEnabled", mad_dump_linkspeeden}, - {BITSOFFS(288, 4), "NeighborMTU", mad_dump_mtu}, - {BITSOFFS(292, 4), "SMSL", mad_dump_uint}, - {BITSOFFS(296, 4), "VLCap", mad_dump_vlcap}, - {BITSOFFS(300, 4), "InitType", mad_dump_hex}, - {BITSOFFS(304, 8), "VLHighLimit", mad_dump_uint}, - {BITSOFFS(312, 8), "VLArbHighCap", mad_dump_uint}, - {BITSOFFS(320, 8), "VLArbLowCap", mad_dump_uint}, - {BITSOFFS(328, 4), "InitReply", mad_dump_hex}, - {BITSOFFS(332, 4), "MtuCap", mad_dump_mtu}, - {BITSOFFS(336, 3), "VLStallCount", mad_dump_uint}, - {BITSOFFS(339, 5), "HoqLife", mad_dump_uint}, - {BITSOFFS(344, 4), "OperVLs", mad_dump_opervls}, - {BITSOFFS(348, 1), "PartEnforceInb", mad_dump_uint}, - {BITSOFFS(349, 1), "PartEnforceOutb", mad_dump_uint}, - {BITSOFFS(350, 1), "FilterRawInb", mad_dump_uint}, - {BITSOFFS(351, 1), "FilterRawOutb", mad_dump_uint}, - {BITSOFFS(352, 16), "MkeyViolations", mad_dump_uint}, - {BITSOFFS(368, 16), "PkeyViolations", mad_dump_uint}, - {BITSOFFS(384, 16), "QkeyViolations", mad_dump_uint}, - {BITSOFFS(400, 8), "GuidCap", mad_dump_uint}, - {BITSOFFS(408, 1), "ClientReregister", mad_dump_uint}, - {BITSOFFS(411, 5), "SubnetTimeout", mad_dump_uint}, - {BITSOFFS(419, 5), "RespTimeVal", mad_dump_uint}, - {BITSOFFS(424, 4), "LocalPhysErr", mad_dump_uint}, - {BITSOFFS(428, 4), "OverrunErr", mad_dump_uint}, - {BITSOFFS(432, 16), "MaxCreditHint", mad_dump_uint}, - {BITSOFFS(456, 24), "RoundTrip", mad_dump_uint}, - {0, 0}, /* IB_PORT_LAST_F */ + [ IB_PORT_MKEY_F ] {0, 64, "Mkey", mad_dump_hex}, + [ IB_PORT_GID_PREFIX_F ] {64, 64, "GidPrefix", mad_dump_hex}, + [ IB_PORT_LID_F ] {BITSOFFS(128, 16), "Lid", mad_dump_uint}, + [ IB_PORT_SMLID_F ] {BITSOFFS(144, 16), "SMLid", mad_dump_uint}, + [ IB_PORT_CAPMASK_F ] {160, 32, "CapMask", mad_dump_portcapmask}, + [ IB_PORT_DIAG_F ] {BITSOFFS(192, 16), "DiagCode", mad_dump_hex}, + [ IB_PORT_MKEY_LEASE_F ] {BITSOFFS(208, 16), "MkeyLeasePeriod", mad_dump_uint}, + [ IB_PORT_LOCAL_PORT_F ] {BITSOFFS(224, 8), "LocalPort", mad_dump_uint}, + [ IB_PORT_LINK_WIDTH_ENABLED_F ] {BITSOFFS(232, 8), "LinkWidthEnabled", mad_dump_linkwidthen}, + [ IB_PORT_LINK_WIDTH_SUPPORTED_F ] {BITSOFFS(240, 8), "LinkWidthSupported", mad_dump_linkwidthsup}, + [ IB_PORT_LINK_WIDTH_ACTIVE_F ] {BITSOFFS(248, 8), "LinkWidthActive", mad_dump_linkwidth}, + [ IB_PORT_LINK_SPEED_SUPPORTED_F ] {BITSOFFS(256, 4), "LinkSpeedSupported", mad_dump_linkspeedsup}, + [ IB_PORT_STATE_F ] {BITSOFFS(260, 4), "LinkState", mad_dump_portstate}, + [ IB_PORT_PHYS_STATE_F ] {BITSOFFS(264, 4), "PhysLinkState", mad_dump_physportstate}, + [ IB_PORT_LINK_DOWN_DEF_F ] {BITSOFFS(268, 4), "LinkDownDefState", mad_dump_linkdowndefstate}, + [ IB_PORT_MKEY_PROT_BITS_F ] {BITSOFFS(272, 2), "ProtectBits", mad_dump_uint}, + [ IB_PORT_LMC_F ] {BITSOFFS(277, 3), "LMC", mad_dump_uint}, + [ IB_PORT_LINK_SPEED_ACTIVE_F ] {BITSOFFS(280, 4), "LinkSpeedActive", mad_dump_linkspeed}, + [ IB_PORT_LINK_SPEED_ENABLED_F ] {BITSOFFS(284, 4), "LinkSpeedEnabled", mad_dump_linkspeeden}, + [ IB_PORT_NEIGHBOR_MTU_F ] {BITSOFFS(288, 4), "NeighborMTU", mad_dump_mtu}, + [ IB_PORT_SMSL_F ] {BITSOFFS(292, 4), "SMSL", mad_dump_uint}, + [ IB_PORT_VL_CAP_F ] {BITSOFFS(296, 4), "VLCap", mad_dump_vlcap}, + [ IB_PORT_INIT_TYPE_F ] {BITSOFFS(300, 4), "InitType", mad_dump_hex}, + [ IB_PORT_VL_HIGH_LIMIT_F ] {BITSOFFS(304, 8), "VLHighLimit", mad_dump_uint}, + [ IB_PORT_VL_ARBITRATION_HIGH_CAP_F ] {BITSOFFS(312, 8), "VLArbHighCap", mad_dump_uint}, + [ IB_PORT_VL_ARBITRATION_LOW_CAP_F ] {BITSOFFS(320, 8), "VLArbLowCap", mad_dump_uint}, + [ IB_PORT_INIT_TYPE_REPLY_F ] {BITSOFFS(328, 4), "InitReply", mad_dump_hex}, + [ IB_PORT_MTU_CAP_F ] {BITSOFFS(332, 4), "MtuCap", mad_dump_mtu}, + [ IB_PORT_VL_STALL_COUNT_F ] {BITSOFFS(336, 3), "VLStallCount", mad_dump_uint}, + [ IB_PORT_HOQ_LIFE_F ] {BITSOFFS(339, 5), "HoqLife", mad_dump_uint}, + [ IB_PORT_OPER_VLS_F ] {BITSOFFS(344, 4), "OperVLs", mad_dump_opervls}, + [ IB_PORT_PART_EN_INB_F ] {BITSOFFS(348, 1), "PartEnforceInb", mad_dump_uint}, + [ IB_PORT_PART_EN_OUTB_F ] {BITSOFFS(349, 1), "PartEnforceOutb", mad_dump_uint}, + [ IB_PORT_FILTER_RAW_INB_F ] {BITSOFFS(350, 1), "FilterRawInb", mad_dump_uint}, + [ IB_PORT_FILTER_RAW_OUTB_F ] {BITSOFFS(351, 1), "FilterRawOutb", mad_dump_uint}, + [ IB_PORT_MKEY_VIOL_F ] {BITSOFFS(352, 16), "MkeyViolations", mad_dump_uint}, + [ IB_PORT_PKEY_VIOL_F ] {BITSOFFS(368, 16), "PkeyViolations", mad_dump_uint}, + [ IB_PORT_QKEY_VIOL_F ] {BITSOFFS(384, 16), "QkeyViolations", mad_dump_uint}, + [ IB_PORT_GUID_CAP_F ] {BITSOFFS(400, 8), "GuidCap", mad_dump_uint}, + [ IB_PORT_CLIENT_REREG_F ] {BITSOFFS(408, 1), "ClientReregister", mad_dump_uint}, + [ IB_PORT_SUBN_TIMEOUT_F ] {BITSOFFS(411, 5), "SubnetTimeout", mad_dump_uint}, + [ IB_PORT_RESP_TIME_VAL_F ] {BITSOFFS(419, 5), "RespTimeVal", mad_dump_uint}, + [ IB_PORT_LOCAL_PHYS_ERR_F ] {BITSOFFS(424, 4), "LocalPhysErr", mad_dump_uint}, + [ IB_PORT_OVERRUN_ERR_F ] {BITSOFFS(428, 4), "OverrunErr", mad_dump_uint}, + [ IB_PORT_MAX_CREDIT_HINT_F ] {BITSOFFS(432, 16), "MaxCreditHint", mad_dump_uint}, + [ IB_PORT_LINK_ROUND_TRIP_F ] {BITSOFFS(456, 24), "RoundTrip", mad_dump_uint}, + [ IB_PORT_LAST_F ] {0, 0}, /* IB_PORT_LAST_F */ /* * NodeInfo fields */ - {BITSOFFS(0, 8), "BaseVers", mad_dump_uint}, - {BITSOFFS(8, 8), "ClassVers", mad_dump_uint}, - {BITSOFFS(16, 8), "NodeType", mad_dump_node_type}, - {BITSOFFS(24, 8), "NumPorts", mad_dump_uint}, - {32, 64, "SystemGuid", mad_dump_hex}, - {96, 64, "Guid", mad_dump_hex}, - {160, 64, "PortGuid", mad_dump_hex}, - {BITSOFFS(224, 16), "PartCap", mad_dump_uint}, - {BITSOFFS(240, 16), "DevId", mad_dump_hex}, - {256, 32, "Revision", mad_dump_hex}, - {BITSOFFS(288, 8), "LocalPort", mad_dump_uint}, - {BITSOFFS(296, 24), "VendorId", mad_dump_hex}, - {0, 0}, /* IB_NODE_LAST_F */ + [ IB_NODE_BASE_VERS_F ] {BITSOFFS(0, 8), "BaseVers", mad_dump_uint}, + [ IB_NODE_CLASS_VERS_F ] {BITSOFFS(8, 8), "ClassVers", mad_dump_uint}, + [ IB_NODE_TYPE_F ] {BITSOFFS(16, 8), "NodeType", mad_dump_node_type}, + [ IB_NODE_NPORTS_F ] {BITSOFFS(24, 8), "NumPorts", mad_dump_uint}, + [ IB_NODE_SYSTEM_GUID_F ] {32, 64, "SystemGuid", mad_dump_hex}, + [ IB_NODE_GUID_F ] {96, 64, "Guid", mad_dump_hex}, + [ IB_NODE_PORT_GUID_F ] {160, 64, "PortGuid", mad_dump_hex}, + [ IB_NODE_PARTITION_CAP_F ] {BITSOFFS(224, 16), "PartCap", mad_dump_uint}, + [ IB_NODE_DEVID_F ] {BITSOFFS(240, 16), "DevId", mad_dump_hex}, + [ IB_NODE_REVISION_F ] {256, 32, "Revision", mad_dump_hex}, + [ IB_NODE_LOCAL_PORT_F ] {BITSOFFS(288, 8), "LocalPort", mad_dump_uint}, + [ IB_NODE_VENDORID_F ] {BITSOFFS(296, 24), "VendorId", mad_dump_hex}, + [ IB_NODE_LAST_F ] {0, 0}, /* IB_NODE_LAST_F */ /* * SwitchInfo fields */ - {BITSOFFS(0, 16), "LinearFdbCap", mad_dump_uint}, - {BITSOFFS(16, 16), "RandomFdbCap", mad_dump_uint}, - {BITSOFFS(32, 16), "McastFdbCap", mad_dump_uint}, - {BITSOFFS(48, 16), "LinearFdbTop", mad_dump_uint}, - {BITSOFFS(64, 8), "DefPort", mad_dump_uint}, - {BITSOFFS(72, 8), "DefMcastPrimPort", mad_dump_uint}, - {BITSOFFS(80, 8), "DefMcastNotPrimPort", mad_dump_uint}, - {BITSOFFS(88, 5), "LifeTime", mad_dump_uint}, - {BITSOFFS(93, 1), "StateChange", mad_dump_uint}, - {BITSOFFS(96, 16), "LidsPerPort", mad_dump_uint}, - {BITSOFFS(112, 16), "PartEnforceCap", mad_dump_uint}, - {BITSOFFS(128, 1), "InboundPartEnf", mad_dump_uint}, - {BITSOFFS(129, 1), "OutboundPartEnf", mad_dump_uint}, - {BITSOFFS(130, 1), "FilterRawInbound", mad_dump_uint}, - {BITSOFFS(131, 1), "FilterRawOutbound", mad_dump_uint}, - {BITSOFFS(132, 1), "EnhancedPort0", mad_dump_uint}, - {0, 0}, /* IB_SW_LAST_F */ + [ IB_SW_LINEAR_FDB_CAP_F ] {BITSOFFS(0, 16), "LinearFdbCap", mad_dump_uint}, + [ IB_SW_RANDOM_FDB_CAP_F ] {BITSOFFS(16, 16), "RandomFdbCap", mad_dump_uint}, + [ IB_SW_MCAST_FDB_CAP_F ] {BITSOFFS(32, 16), "McastFdbCap", mad_dump_uint}, + [ IB_SW_LINEAR_FDB_TOP_F ] {BITSOFFS(48, 16), "LinearFdbTop", mad_dump_uint}, + [ IB_SW_DEF_PORT_F ] {BITSOFFS(64, 8), "DefPort", mad_dump_uint}, + [ IB_SW_DEF_MCAST_PRIM_F ] {BITSOFFS(72, 8), "DefMcastPrimPort", mad_dump_uint}, + [ IB_SW_DEF_MCAST_NOT_PRIM_F ] {BITSOFFS(80, 8), "DefMcastNotPrimPort", mad_dump_uint}, + [ IB_SW_LIFE_TIME_F ] {BITSOFFS(88, 5), "LifeTime", mad_dump_uint}, + [ IB_SW_STATE_CHANGE_F ] {BITSOFFS(93, 1), "StateChange", mad_dump_uint}, + [ IB_SW_LIDS_PER_PORT_F ] {BITSOFFS(96, 16), "LidsPerPort", mad_dump_uint}, + [ IB_SW_PARTITION_ENFORCE_CAP_F ] {BITSOFFS(112, 16), "PartEnforceCap", mad_dump_uint}, + [ IB_SW_PARTITION_ENF_INB_F ] {BITSOFFS(128, 1), "InboundPartEnf", mad_dump_uint}, + [ IB_SW_PARTITION_ENF_OUTB_F ] {BITSOFFS(129, 1), "OutboundPartEnf", mad_dump_uint}, + [ IB_SW_FILTER_RAW_INB_F ] {BITSOFFS(130, 1), "FilterRawInbound", mad_dump_uint}, + [ IB_SW_FILTER_RAW_OUTB_F ] {BITSOFFS(131, 1), "FilterRawOutbound", mad_dump_uint}, + [ IB_SW_ENHANCED_PORT0_F ] {BITSOFFS(132, 1), "EnhancedPort0", mad_dump_uint}, + [ IB_SW_LAST_F ] {0, 0}, /* IB_SW_LAST_F */ /* * SwitchLinearForwardingTable fields */ - {0, 512, "LinearForwTbl", mad_dump_array}, + [ IB_LINEAR_FORW_TBL_F ] {0, 512, "LinearForwTbl", mad_dump_array}, /* * SwitchMulticastForwardingTable fields */ - {0, 512, "MulticastForwTbl", mad_dump_array}, + [ IB_MULTICAST_FORW_TBL_F ] {0, 512, "MulticastForwTbl", mad_dump_array}, /* * NodeDescription fields */ - {0, 64 * 8, "NodeDesc", mad_dump_string}, + [ IB_NODE_DESC_F ] {0, 64 * 8, "NodeDesc", mad_dump_string}, /* * Notice/Trap fields */ - {BITSOFFS(0, 1), "NoticeIsGeneric", mad_dump_uint}, - {BITSOFFS(1, 7), "NoticeType", mad_dump_uint}, - {BITSOFFS(8, 24), "NoticeProducerType", mad_dump_node_type}, - {BITSOFFS(32, 16), "NoticeTrapNumber", mad_dump_uint}, - {BITSOFFS(48, 16), "NoticeIssuerLID", mad_dump_uint}, - {BITSOFFS(64, 1), "NoticeToggle", mad_dump_uint}, - {BITSOFFS(65, 15), "NoticeCount", mad_dump_uint}, - {80, 432, "NoticeDataDetails", mad_dump_array}, - {BITSOFFS(80, 16), "NoticeDataLID", mad_dump_uint}, - {BITSOFFS(96, 16), "NoticeDataTrap144LID", mad_dump_uint}, - {BITSOFFS(128, 32), "NoticeDataTrap144CapMask", mad_dump_uint}, + [ IB_NOTICE_IS_GENERIC_F ] {BITSOFFS(0, 1), "NoticeIsGeneric", mad_dump_uint}, + [ IB_NOTICE_TYPE_F ] {BITSOFFS(1, 7), "NoticeType", mad_dump_uint}, + [ IB_NOTICE_PRODUCER_F ] {BITSOFFS(8, 24), "NoticeProducerType", mad_dump_node_type}, + [ IB_NOTICE_TRAP_NUMBER_F ] {BITSOFFS(32, 16), "NoticeTrapNumber", mad_dump_uint}, + [ IB_NOTICE_ISSUER_LID_F ] {BITSOFFS(48, 16), "NoticeIssuerLID", mad_dump_uint}, + [ IB_NOTICE_TOGGLE_F ] {BITSOFFS(64, 1), "NoticeToggle", mad_dump_uint}, + [ IB_NOTICE_COUNT_F ] {BITSOFFS(65, 15), "NoticeCount", mad_dump_uint}, + [ IB_NOTICE_DATA_DETAILS_F ] {80, 432, "NoticeDataDetails", mad_dump_array}, + [ IB_NOTICE_DATA_LID_F ] {BITSOFFS(80, 16), "NoticeDataLID", mad_dump_uint}, + [ IB_NOTICE_DATA_144_LID_F ] {BITSOFFS(96, 16), "NoticeDataTrap144LID", mad_dump_uint}, + [ IB_NOTICE_DATA_144_CAPMASK_F ] {BITSOFFS(128, 32), "NoticeDataTrap144CapMask", mad_dump_uint}, /* * Port counters */ - {BITSOFFS(8, 8), "PortSelect", mad_dump_uint}, - {BITSOFFS(16, 16), "CounterSelect", mad_dump_hex}, - {BITSOFFS(32, 16), "SymbolErrors", mad_dump_uint}, - {BITSOFFS(48, 8), "LinkRecovers", mad_dump_uint}, - {BITSOFFS(56, 8), "LinkDowned", mad_dump_uint}, - {BITSOFFS(64, 16), "RcvErrors", mad_dump_uint}, - {BITSOFFS(80, 16), "RcvRemotePhysErrors", mad_dump_uint}, - {BITSOFFS(96, 16), "RcvSwRelayErrors", mad_dump_uint}, - {BITSOFFS(112, 16), "XmtDiscards", mad_dump_uint}, - {BITSOFFS(128, 8), "XmtConstraintErrors", mad_dump_uint}, - {BITSOFFS(136, 8), "RcvConstraintErrors", mad_dump_uint}, - {BITSOFFS(144, 8), "CounterSelect2", mad_dump_uint}, - {BITSOFFS(152, 4), "LinkIntegrityErrors", mad_dump_uint}, - {BITSOFFS(156, 4), "ExcBufOverrunErrors", mad_dump_uint}, - {BITSOFFS(176, 16), "VL15Dropped", mad_dump_uint}, - {192, 32, "XmtData", mad_dump_uint}, - {224, 32, "RcvData", mad_dump_uint}, - {256, 32, "XmtPkts", mad_dump_uint}, - {288, 32, "RcvPkts", mad_dump_uint}, - {320, 32, "XmtWait", mad_dump_uint}, - {0, 0}, /* IB_PC_LAST_F */ + [ IB_PC_PORT_SELECT_F ] {BITSOFFS(8, 8), "PortSelect", mad_dump_uint}, + [ IB_PC_COUNTER_SELECT_F ] {BITSOFFS(16, 16), "CounterSelect", mad_dump_hex}, + [ IB_PC_ERR_SYM_F ] {BITSOFFS(32, 16), "SymbolErrors", mad_dump_uint}, + [ IB_PC_LINK_RECOVERS_F ] {BITSOFFS(48, 8), "LinkRecovers", mad_dump_uint}, + [ IB_PC_LINK_DOWNED_F ] {BITSOFFS(56, 8), "LinkDowned", mad_dump_uint}, + [ IB_PC_ERR_RCV_F ] {BITSOFFS(64, 16), "RcvErrors", mad_dump_uint}, + [ IB_PC_ERR_PHYSRCV_F ] {BITSOFFS(80, 16), "RcvRemotePhysErrors", mad_dump_uint}, + [ IB_PC_ERR_SWITCH_REL_F ] {BITSOFFS(96, 16), "RcvSwRelayErrors", mad_dump_uint}, + [ IB_PC_XMT_DISCARDS_F ] {BITSOFFS(112, 16), "XmtDiscards", mad_dump_uint}, + [ IB_PC_ERR_XMTCONSTR_F ] {BITSOFFS(128, 8), "XmtConstraintErrors", mad_dump_uint}, + [ IB_PC_ERR_RCVCONSTR_F ] {BITSOFFS(136, 8), "RcvConstraintErrors", mad_dump_uint}, + [ IB_PC_COUNTER_SELECT2_F ] {BITSOFFS(144, 8), "CounterSelect2", mad_dump_uint}, + [ IB_PC_ERR_LOCALINTEG_F ] {BITSOFFS(152, 4), "LinkIntegrityErrors", mad_dump_uint}, + [ IB_PC_ERR_EXCESS_OVR_F ] {BITSOFFS(156, 4), "ExcBufOverrunErrors", mad_dump_uint}, + [ IB_PC_VL15_DROPPED_F ] {BITSOFFS(176, 16), "VL15Dropped", mad_dump_uint}, + [ IB_PC_XMT_BYTES_F ] {192, 32, "XmtData", mad_dump_uint}, + [ IB_PC_RCV_BYTES_F ] {224, 32, "RcvData", mad_dump_uint}, + [ IB_PC_XMT_PKTS_F ] {256, 32, "XmtPkts", mad_dump_uint}, + [ IB_PC_RCV_PKTS_F ] {288, 32, "RcvPkts", mad_dump_uint}, + [ IB_PC_XMT_WAIT_F ] {320, 32, "XmtWait", mad_dump_uint}, + [ IB_PC_LAST_F ] {0, 0}, /* IB_PC_LAST_F */ /* * SMInfo */ - {0, 64, "SmInfoGuid", mad_dump_hex}, - {64, 64, "SmInfoKey", mad_dump_hex}, - {128, 32, "SmActivity", mad_dump_uint}, - {BITSOFFS(160, 4), "SmPriority", mad_dump_uint}, - {BITSOFFS(164, 4), "SmState", mad_dump_uint}, + [ IB_SMINFO_GUID_F ] {0, 64, "SmInfoGuid", mad_dump_hex}, + [ IB_SMINFO_KEY_F ] {64, 64, "SmInfoKey", mad_dump_hex}, + [ IB_SMINFO_ACT_F ] {128, 32, "SmActivity", mad_dump_uint}, + [ IB_SMINFO_PRIO_F ] {BITSOFFS(160, 4), "SmPriority", mad_dump_uint}, + [ IB_SMINFO_STATE_F ] {BITSOFFS(164, 4), "SmState", mad_dump_uint}, /* * SA RMPP */ - {BE_OFFS(24 * 8 + 24, 8), "RmppVers", mad_dump_uint}, - {BE_OFFS(24 * 8 + 16, 8), "RmppType", mad_dump_uint}, - {BE_OFFS(24 * 8 + 11, 5), "RmppResp", mad_dump_uint}, - {BE_OFFS(24 * 8 + 8, 3), "RmppFlags", mad_dump_hex}, - {BE_OFFS(24 * 8 + 0, 8), "RmppStatus", mad_dump_hex}, + [ IB_SA_RMPP_VERS_F ] {BE_OFFS(24 * 8 + 24, 8), "RmppVers", mad_dump_uint}, + [ IB_SA_RMPP_TYPE_F ] {BE_OFFS(24 * 8 + 16, 8), "RmppType", mad_dump_uint}, + [ IB_SA_RMPP_RESP_F ] {BE_OFFS(24 * 8 + 11, 5), "RmppResp", mad_dump_uint}, + [ IB_SA_RMPP_FLAGS_F ] {BE_OFFS(24 * 8 + 8, 3), "RmppFlags", mad_dump_hex}, + [ IB_SA_RMPP_STATUS_F ] {BE_OFFS(24 * 8 + 0, 8), "RmppStatus", mad_dump_hex}, /* data1 */ - {28 * 8, 32, "RmppData1", mad_dump_hex}, - {28 * 8, 32, "RmppSegNum", mad_dump_uint}, + [ IB_SA_RMPP_D1_F ] {28 * 8, 32, "RmppData1", mad_dump_hex}, + [ IB_SA_RMPP_SEGNUM_F ] {28 * 8, 32, "RmppSegNum", mad_dump_uint}, /* data2 */ - {32 * 8, 32, "RmppData2", mad_dump_hex}, - {32 * 8, 32, "RmppPayload", mad_dump_uint}, - {32 * 8, 32, "RmppNewWin", mad_dump_uint}, + [ IB_SA_RMPP_D2_F ] {32 * 8, 32, "RmppData2", mad_dump_hex}, + [ IB_SA_RMPP_LEN_F ] {32 * 8, 32, "RmppPayload", mad_dump_uint}, + [ IB_SA_RMPP_NEWWIN_F ] {32 * 8, 32, "RmppNewWin", mad_dump_uint}, /* * SA Get Multi Path */ - {BITSOFFS(41, 7), "MultiPathNumPath", mad_dump_uint}, - {BITSOFFS(120, 8), "MultiPathNumSrc", mad_dump_uint}, - {BITSOFFS(128, 8), "MultiPathNumDest", mad_dump_uint}, - {192, 128, "MultiPathGid", mad_dump_array}, + [ IB_SA_MP_NPATH_F ] {BITSOFFS(41, 7), "MultiPathNumPath", mad_dump_uint}, + [ IB_SA_MP_NSRC_F ] {BITSOFFS(120, 8), "MultiPathNumSrc", mad_dump_uint}, + [ IB_SA_MP_NDEST_F ] {BITSOFFS(128, 8), "MultiPathNumDest", mad_dump_uint}, + [ IB_SA_MP_GID0_F ] {192, 128, "MultiPathGid", mad_dump_array}, /* * SA Path rec */ - {64, 128, "PathRecDGid", mad_dump_array}, - {192, 128, "PathRecSGid", mad_dump_array}, - {BITSOFFS(320, 16), "PathRecDLid", mad_dump_hex}, - {BITSOFFS(336, 16), "PathRecSLid", mad_dump_hex}, - {BITSOFFS(393, 7), "PathRecNumPath", mad_dump_uint}, + [ IB_SA_PR_DGID_F ] {64, 128, "PathRecDGid", mad_dump_array}, + [ IB_SA_PR_SGID_F ] {192, 128, "PathRecSGid", mad_dump_array}, + [ IB_SA_PR_DLID_F ] {BITSOFFS(320, 16), "PathRecDLid", mad_dump_hex}, + [ IB_SA_PR_SLID_F ] {BITSOFFS(336, 16), "PathRecSLid", mad_dump_hex}, + [ IB_SA_PR_NPATH_F ] {BITSOFFS(393, 7), "PathRecNumPath", mad_dump_uint}, /* * MC Member rec */ - {0, 128, "McastMemMGid", mad_dump_array}, - {128, 128, "McastMemPortGid", mad_dump_array}, - {256, 32, "McastMemQkey", mad_dump_hex}, - {BITSOFFS(288, 16), "McastMemMLid", mad_dump_hex}, - {BITSOFFS(352, 4), "McastMemSL", mad_dump_uint}, - {BITSOFFS(306, 6), "McastMemMTU", mad_dump_uint}, - {BITSOFFS(338, 6), "McastMemRate", mad_dump_uint}, - {BITSOFFS(312, 8), "McastMemTClass", mad_dump_uint}, - {BITSOFFS(320, 16), "McastMemPkey", mad_dump_uint}, - {BITSOFFS(356, 20), "McastMemFlowLbl", mad_dump_uint}, - {BITSOFFS(388, 4), "McastMemJoinState", mad_dump_uint}, - {BITSOFFS(392, 1), "McastMemProxyJoin", mad_dump_uint}, + [ IB_SA_MCM_MGID_F ] {0, 128, "McastMemMGid", mad_dump_array}, + [ IB_SA_MCM_PORTGID_F ] {128, 128, "McastMemPortGid", mad_dump_array}, + [ IB_SA_MCM_QKEY_F ] {256, 32, "McastMemQkey", mad_dump_hex}, + [ IB_SA_MCM_MLID_F ] {BITSOFFS(288, 16), "McastMemMLid", mad_dump_hex}, + [ IB_SA_MCM_SL_F ] {BITSOFFS(352, 4), "McastMemSL", mad_dump_uint}, + [ IB_SA_MCM_MTU_F ] {BITSOFFS(306, 6), "McastMemMTU", mad_dump_uint}, + [ IB_SA_MCM_RATE_F ] {BITSOFFS(338, 6), "McastMemRate", mad_dump_uint}, + [ IB_SA_MCM_TCLASS_F ] {BITSOFFS(312, 8), "McastMemTClass", mad_dump_uint}, + [ IB_SA_MCM_PKEY_F ] {BITSOFFS(320, 16), "McastMemPkey", mad_dump_uint}, + [ IB_SA_MCM_FLOW_LABEL_F ] {BITSOFFS(356, 20), "McastMemFlowLbl", mad_dump_uint}, + [ IB_SA_MCM_JOIN_STATE_F ] {BITSOFFS(388, 4), "McastMemJoinState", mad_dump_uint}, + [ IB_SA_MCM_PROXY_JOIN_F ] {BITSOFFS(392, 1), "McastMemProxyJoin", mad_dump_uint}, /* * Service record */ - {0, 64, "ServRecID", mad_dump_hex}, - {64, 128, "ServRecGid", mad_dump_array}, - {BITSOFFS(192, 16), "ServRecPkey", mad_dump_hex}, - {224, 32, "ServRecLease", mad_dump_hex}, - {256, 128, "ServRecKey", mad_dump_hex}, - {384, 512, "ServRecName", mad_dump_string}, - {896, 512, "ServRecData", mad_dump_array}, /* ATS for example */ + [ IB_SA_SR_ID_F ] {0, 64, "ServRecID", mad_dump_hex}, + [ IB_SA_SR_GID_F ] {64, 128, "ServRecGid", mad_dump_array}, + [ IB_SA_SR_PKEY_F ] {BITSOFFS(192, 16), "ServRecPkey", mad_dump_hex}, + [ IB_SA_SR_LEASE_F ] {224, 32, "ServRecLease", mad_dump_hex}, + [ IB_SA_SR_KEY_F ] {256, 128, "ServRecKey", mad_dump_hex}, + [ IB_SA_SR_NAME_F ] {384, 512, "ServRecName", mad_dump_string}, + [ IB_SA_SR_DATA_F ] {896, 512, "ServRecData", mad_dump_array}, /* ATS for example */ /* * ATS SM record - within SA_SR_DATA */ - {12 * 8, 32, "ATSNodeAddr", mad_dump_hex}, - {BITSOFFS(16 * 8, 16), "ATSMagicKey", mad_dump_hex}, - {BITSOFFS(18 * 8, 16), "ATSNodeType", mad_dump_hex}, - {32 * 8, 32 * 8, "ATSNodeName", mad_dump_string}, + [ IB_ATS_SM_NODE_ADDR_F ] {12 * 8, 32, "ATSNodeAddr", mad_dump_hex}, + [ IB_ATS_SM_MAGIC_KEY_F ] {BITSOFFS(16 * 8, 16), "ATSMagicKey", mad_dump_hex}, + [ IB_ATS_SM_NODE_TYPE_F ] {BITSOFFS(18 * 8, 16), "ATSNodeType", mad_dump_hex}, + [ IB_ATS_SM_NODE_NAME_F ] {32 * 8, 32 * 8, "ATSNodeName", mad_dump_string}, /* * SLTOVL MAPPING TABLE */ - {0, 64, "SLToVLMap", mad_dump_hex}, + [ IB_SLTOVL_MAPPING_TABLE_F ] {0, 64, "SLToVLMap", mad_dump_hex}, /* * VL ARBITRATION TABLE */ - {0, 512, "VLArbTbl", mad_dump_array}, + [ IB_VL_ARBITRATION_TABLE_F ] {0, 512, "VLArbTbl", mad_dump_array}, /* * IB vendor classes range 2 */ - {BE_OFFS(36 * 8, 24), "OUI", mad_dump_array}, - {40 * 8, (256 - 40) * 8, "Vendor2Data", mad_dump_array}, + [ IB_VEND2_OUI_F ] {BE_OFFS(36 * 8, 24), "OUI", mad_dump_array}, + [ IB_VEND2_DATA_F ] {40 * 8, (256 - 40) * 8, "Vendor2Data", mad_dump_array}, /* * Extended port counters */ - {BITSOFFS(8, 8), "PortSelect", mad_dump_uint}, - {BITSOFFS(16, 16), "CounterSelect", mad_dump_hex}, - {64, 64, "PortXmitData", mad_dump_uint}, - {128, 64, "PortRcvData", mad_dump_uint}, - {192, 64, "PortXmitPkts", mad_dump_uint}, - {256, 64, "PortRcvPkts", mad_dump_uint}, - {320, 64, "PortUnicastXmitPkts", mad_dump_uint}, - {384, 64, "PortUnicastRcvPkts", mad_dump_uint}, - {448, 64, "PortMulticastXmitPkts", mad_dump_uint}, - {512, 64, "PortMulticastRcvPkts", mad_dump_uint}, - {0, 0}, /* IB_PC_EXT_LAST_F */ + [ IB_PC_EXT_PORT_SELECT_F ] {BITSOFFS(8, 8), "PortSelect", mad_dump_uint}, + [ IB_PC_EXT_COUNTER_SELECT_F ] {BITSOFFS(16, 16), "CounterSelect", mad_dump_hex}, + [ IB_PC_EXT_XMT_BYTES_F ] {64, 64, "PortXmitData", mad_dump_uint}, + [ IB_PC_EXT_RCV_BYTES_F ] {128, 64, "PortRcvData", mad_dump_uint}, + [ IB_PC_EXT_XMT_PKTS_F ] {192, 64, "PortXmitPkts", mad_dump_uint}, + [ IB_PC_EXT_RCV_PKTS_F ] {256, 64, "PortRcvPkts", mad_dump_uint}, + [ IB_PC_EXT_XMT_UPKTS_F ] {320, 64, "PortUnicastXmitPkts", mad_dump_uint}, + [ IB_PC_EXT_RCV_UPKTS_F ] {384, 64, "PortUnicastRcvPkts", mad_dump_uint}, + [ IB_PC_EXT_XMT_MPKTS_F ] {448, 64, "PortMulticastXmitPkts", mad_dump_uint}, + [ IB_PC_EXT_RCV_MPKTS_F ] {512, 64, "PortMulticastRcvPkts", mad_dump_uint}, + [ IB_PC_EXT_LAST_F ] {0, 0}, /* IB_PC_EXT_LAST_F */ /* * GUIDInfo fields */ - {0, 64, "GUID0", mad_dump_hex}, + [ IB_GUID_GUID0_F ] {0, 64, "GUID0", mad_dump_hex}, /* * ClassPortInfo fields */ - {BITSOFFS(0, 8), "BaseVersion", mad_dump_uint}, - {BITSOFFS(8, 8), "ClassVersion", mad_dump_uint}, - {BITSOFFS(16, 16), "CapabilityMask", mad_dump_hex}, - {BITSOFFS(32, 27), "CapabilityMask2", mad_dump_hex}, - {BITSOFFS(59, 5), "RespTimeVal", mad_dump_uint}, - {64, 128, "RedirectGID", mad_dump_array}, - {BITSOFFS(192, 8), "RedirectTC", mad_dump_hex}, - {BITSOFFS(200, 4), "RedirectSL", mad_dump_uint}, - {BITSOFFS(204, 20), "RedirectFL", mad_dump_hex}, - {BITSOFFS(224, 16), "RedirectLID", mad_dump_hex}, - {BITSOFFS(240, 16), "RedirectPKey", mad_dump_hex}, - {BITSOFFS(264, 24), "RedirectQP", mad_dump_hex}, - {288, 32, "RedirectQKey", mad_dump_hex}, - {320, 128, "TrapGID", mad_dump_array}, - {BITSOFFS(448, 8), "TrapTC", mad_dump_hex}, - {BITSOFFS(456, 4), "TrapSL", mad_dump_uint}, - {BITSOFFS(460, 20), "TrapFL", mad_dump_hex}, - {BITSOFFS(480, 16), "TrapLID", mad_dump_hex}, - {BITSOFFS(496, 16), "TrapPKey", mad_dump_hex}, - {BITSOFFS(512, 8), "TrapHL", mad_dump_uint}, - {BITSOFFS(520, 24), "TrapQP", mad_dump_hex}, - {544, 32, "TrapQKey", mad_dump_hex}, - - {32, 32, "XmtDataSL0", mad_dump_uint}, - {64, 32, "XmtDataSL1", mad_dump_uint}, - {96, 32, "XmtDataSL2", mad_dump_uint}, - {128, 32, "XmtDataSL3", mad_dump_uint}, - {160, 32, "XmtDataSL4", mad_dump_uint}, - {192, 32, "XmtDataSL5", mad_dump_uint}, - {224, 32, "XmtDataSL6", mad_dump_uint}, - {256, 32, "XmtDataSL7", mad_dump_uint}, - {288, 32, "XmtDataSL8", mad_dump_uint}, - {320, 32, "XmtDataSL9", mad_dump_uint}, - {352, 32, "XmtDataSL10", mad_dump_uint}, - {384, 32, "XmtDataSL11", mad_dump_uint}, - {416, 32, "XmtDataSL12", mad_dump_uint}, - {448, 32, "XmtDataSL13", mad_dump_uint}, - {480, 32, "XmtDataSL14", mad_dump_uint}, - {512, 32, "XmtDataSL15", mad_dump_uint}, - {0, 0}, /* IB_PC_XMT_DATA_SL_LAST_F */ - - {32, 32, "RcvDataSL0", mad_dump_uint}, - {64, 32, "RcvDataSL1", mad_dump_uint}, - {96, 32, "RcvDataSL2", mad_dump_uint}, - {128, 32, "RcvDataSL3", mad_dump_uint}, - {160, 32, "RcvDataSL4", mad_dump_uint}, - {192, 32, "RcvDataSL5", mad_dump_uint}, - {224, 32, "RcvDataSL6", mad_dump_uint}, - {256, 32, "RcvDataSL7", mad_dump_uint}, - {288, 32, "RcvDataSL8", mad_dump_uint}, - {320, 32, "RcvDataSL9", mad_dump_uint}, - {352, 32, "RcvDataSL10", mad_dump_uint}, - {384, 32, "RcvDataSL11", mad_dump_uint}, - {416, 32, "RcvDataSL12", mad_dump_uint}, - {448, 32, "RcvDataSL13", mad_dump_uint}, - {480, 32, "RcvDataSL14", mad_dump_uint}, - {512, 32, "RcvDataSL15", mad_dump_uint}, - {0, 0}, /* IB_PC_RCV_DATA_SL_LAST_F */ - - {0, 0} /* IB_FIELD_LAST_ */ - + [ IB_CPI_BASEVER_F ] {BITSOFFS(0, 8), "BaseVersion", mad_dump_uint}, + [ IB_CPI_CLASSVER_F ] {BITSOFFS(8, 8), "ClassVersion", mad_dump_uint}, + [ IB_CPI_CAPMASK_F ] {BITSOFFS(16, 16), "CapabilityMask", mad_dump_hex}, + [ IB_CPI_CAPMASK2_F ] {BITSOFFS(32, 27), "CapabilityMask2", mad_dump_hex}, + [ IB_CPI_RESP_TIME_VALUE_F ] {BITSOFFS(59, 5), "RespTimeVal", mad_dump_uint}, + [ IB_CPI_REDIRECT_GID_F ] {64, 128, "RedirectGID", mad_dump_array}, + [ IB_CPI_REDIRECT_TC_F ] {BITSOFFS(192, 8), "RedirectTC", mad_dump_hex}, + [ IB_CPI_REDIRECT_SL_F ] {BITSOFFS(200, 4), "RedirectSL", mad_dump_uint}, + [ IB_CPI_REDIRECT_FL_F ] {BITSOFFS(204, 20), "RedirectFL", mad_dump_hex}, + [ IB_CPI_REDIRECT_LID_F ] {BITSOFFS(224, 16), "RedirectLID", mad_dump_hex}, + [ IB_CPI_REDIRECT_PKEY_F ] {BITSOFFS(240, 16), "RedirectPKey", mad_dump_hex}, + [ IB_CPI_REDIRECT_QP_F ] {BITSOFFS(264, 24), "RedirectQP", mad_dump_hex}, + [ IB_CPI_REDIRECT_QKEY_F ] {288, 32, "RedirectQKey", mad_dump_hex}, + [ IB_CPI_TRAP_GID_F ] {320, 128, "TrapGID", mad_dump_array}, + [ IB_CPI_TRAP_TC_F ] {BITSOFFS(448, 8), "TrapTC", mad_dump_hex}, + [ IB_CPI_TRAP_SL_F ] {BITSOFFS(456, 4), "TrapSL", mad_dump_uint}, + [ IB_CPI_TRAP_FL_F ] {BITSOFFS(460, 20), "TrapFL", mad_dump_hex}, + [ IB_CPI_TRAP_LID_F ] {BITSOFFS(480, 16), "TrapLID", mad_dump_hex}, + [ IB_CPI_TRAP_PKEY_F ] {BITSOFFS(496, 16), "TrapPKey", mad_dump_hex}, + [ IB_CPI_TRAP_HL_F ] {BITSOFFS(512, 8), "TrapHL", mad_dump_uint}, + [ IB_CPI_TRAP_QP_F ] {BITSOFFS(520, 24), "TrapQP", mad_dump_hex}, + [ IB_CPI_TRAP_QKEY_F ] {544, 32, "TrapQKey", mad_dump_hex}, + + [ IB_PC_XMT_DATA_SL0_F ] {32, 32, "XmtDataSL0", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL1_F ] {64, 32, "XmtDataSL1", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL2_F ] {96, 32, "XmtDataSL2", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL3_F ] {128, 32, "XmtDataSL3", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL4_F ] {160, 32, "XmtDataSL4", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL5_F ] {192, 32, "XmtDataSL5", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL6_F ] {224, 32, "XmtDataSL6", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL7_F ] {256, 32, "XmtDataSL7", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL8_F ] {288, 32, "XmtDataSL8", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL9_F ] {320, 32, "XmtDataSL9", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL10_F ] {352, 32, "XmtDataSL10", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL11_F ] {384, 32, "XmtDataSL11", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL12_F ] {416, 32, "XmtDataSL12", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL13_F ] {448, 32, "XmtDataSL13", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL14_F ] {480, 32, "XmtDataSL14", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL15_F ] {512, 32, "XmtDataSL15", mad_dump_uint}, + [ IB_PC_XMT_DATA_SL_LAST_F ] {0, 0}, /* IB_PC_XMT_DATA_SL_LAST_F */ + + [ IB_PC_RCV_DATA_SL0_F ] {32, 32, "RcvDataSL0", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL1_F ] {64, 32, "RcvDataSL1", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL2_F ] {96, 32, "RcvDataSL2", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL3_F ] {128, 32, "RcvDataSL3", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL4_F ] {160, 32, "RcvDataSL4", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL5_F ] {192, 32, "RcvDataSL5", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL6_F ] {224, 32, "RcvDataSL6", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL7_F ] {256, 32, "RcvDataSL7", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL8_F ] {288, 32, "RcvDataSL8", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL9_F ] {320, 32, "RcvDataSL9", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL10_F ] {352, 32, "RcvDataSL10", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL11_F ] {384, 32, "RcvDataSL11", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL12_F ] {416, 32, "RcvDataSL12", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL13_F ] {448, 32, "RcvDataSL13", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL14_F ] {480, 32, "RcvDataSL14", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL15_F ] {512, 32, "RcvDataSL15", mad_dump_uint}, + [ IB_PC_RCV_DATA_SL_LAST_F ] {0, 0}, /* IB_PC_RCV_DATA_SL_LAST_F */ + + [ IB_FIELD_LAST_ ] {0, 0} /* IB_FIELD_LAST_ */ }; static void _set_field64(void *buf, int base_offs, const ib_field_t * f, -- 1.5.5 From hal.rosenstock at gmail.com Wed Mar 18 02:53:25 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 18 Mar 2009 05:53:25 -0400 Subject: ***SPAM*** Re: [ofa-general] 4K MTU with OpenSM? In-Reply-To: <20090318094412.GF604@sashak.voltaire.com> References: <49BFF3D8.4040402@sun.com> <20090318094412.GF604@sashak.voltaire.com> Message-ID: On Wed, Mar 18, 2009 at 5:44 AM, Sasha Khapyorsky wrote: > On 18:17 Tue 17 Mar     , Dave Olson wrote: >> On Tue, 17 Mar 2009, Hal Rosenstock wrote: >> | On Tue, Mar 17, 2009 at 3:02 PM, Ted H. Kim wrote: >> | > Is there way to config 4K IB MTU with OpenSM? >> | > What's the option of file setting for that? >> | >> | See the partition configuration section in the man page or >> | opensm/doc/partition-config.txt. It's something like: >> | >> | Default=0x7fff,ipoib,mtu=5:ALL=full; > > This will use this MTU value for IPoIB multicast group. > > As far as I understood Ted's question was "how to setup port's MTU using > OpenSM". Assuming so I don't think that OpenSM has such functionality > now (although it should not be very hard to add). Do you mean admin the port's MTU ? If so, why would there be such a need ? The only time it might be needed would be to select a lower MTU than the peers support. That was done for Tavor for a performance issue. Do you envision some other need ? -- Hal > Sasha From hal.rosenstock at gmail.com Wed Mar 18 03:02:41 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 18 Mar 2009 06:02:41 -0400 Subject: ***SPAM*** Re: ***SPAM*** Re: [ofa-general] 4K MTU with OpenSM? In-Reply-To: References: <49BFF3D8.4040402@sun.com> <20090318094412.GF604@sashak.voltaire.com> Message-ID: On Wed, Mar 18, 2009 at 5:53 AM, Hal Rosenstock wrote: > On Wed, Mar 18, 2009 at 5:44 AM, Sasha Khapyorsky wrote: >> On 18:17 Tue 17 Mar     , Dave Olson wrote: >>> On Tue, 17 Mar 2009, Hal Rosenstock wrote: >>> | On Tue, Mar 17, 2009 at 3:02 PM, Ted H. Kim wrote: >>> | > Is there way to config 4K IB MTU with OpenSM? >>> | > What's the option of file setting for that? >>> | >>> | See the partition configuration section in the man page or >>> | opensm/doc/partition-config.txt. It's something like: >>> | >>> | Default=0x7fff,ipoib,mtu=5:ALL=full; >> >> This will use this MTU value for IPoIB multicast group. >> >> As far as I understood Ted's question was "how to setup port's MTU using >> OpenSM". Assuming so I don't think that OpenSM has such functionality >> now (although it should not be very hard to add). > > Do you mean admin the port's MTU ? If so, why would there be such a need ? > > The only time it might be needed would be to select a lower MTU than > the peers support. In short, if the peer ports both advertise 4K MTU being supported, then most SMs (including OpenSM) will set the link to run at that. -- Hal > That was done for Tavor for a performance issue. Do > you envision some other need ? > > -- Hal > >> Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From vlad at lists.openfabrics.org Wed Mar 18 03:25:23 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Wed, 18 Mar 2009 03:25:23 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090318-0200 daily build status Message-ID: <20090318102524.1A10FE60DD8@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: Build failed on x86_64 with linux-2.6.18-128.el5 Log: -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Wstrict-prototypes -Wundef -Werror-implicit-function-declaration -Os -m64 -mno-red-zone -mcmodel=kernel -pipe -fno-reorder-blocks -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -fomit-frame-pointer -g -fno-stack-protector -Wdeclaration-after-statement -Wno-pointer-sign -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(af_rds)" -D"KBUILD_MODNAME=KBUILD_STR(rds)" -c -o /home/vlad/tmp/ofa_1_4_kernel-20090318-0200_linux-2.6.18-128.el5_x86_64_check/net/rds/.tmp_af_rds.o /home/vlad/tmp/ofa_1_4_kernel-20090318-0200_linux-2.6.18-128.el5_x86_64_check/net/rds/af_rds.c In file included from /home/vlad/tmp/ofa_1_4_kernel-20090318-0200_linux-2.6.18-128.el5_x86_64_check/net/rds/af_rds.c:41: /home/vlad/tmp/ofa_1_4_kernel-20090318-0200_linux-2.6.18-128.el5_x86_64_check/net/rds/rds.h:12: error: redefinition of typedef '__sum16' /home/vlad/tmp/ofa_1_4_kernel-20090318-0200_linux-2.6.18-128.el5_x86_64_check/kernel_addons/backport/2.6.18-EL5.3/include/linux/types.h:6: error: previous declaration of '__sum16' was here make[3]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090318-0200_linux-2.6.18-128.el5_x86_64_check/net/rds/af_rds.o] Error 1 make[2]: *** [/home/vlad/tmp/ofa_1_4_kernel-20090318-0200_linux-2.6.18-128.el5_x86_64_check/net/rds] Error 2 make[1]: *** [_module_/home/vlad/tmp/ofa_1_4_kernel-20090318-0200_linux-2.6.18-128.el5_x86_64_check] Error 2 make[1]: Leaving directory `/home/vlad/kernel.org/x86_64/linux-2.6.18-128.el5' make: *** [kernel] Error 2 ---------------------------------------------------------------------------------- From hnrose at comcast.net Wed Mar 18 05:18:07 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 18 Mar 2009 07:18:07 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] opensm/osm_helper.c: Add more info for traps 144 and 256-259 in osm_dump_notice Message-ID: <20090318121807.GA5284@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_helper.c b/opensm/opensm/osm_helper.c index b40ba0c..593bbf7 100644 --- a/opensm/opensm/osm_helper.c +++ b/opensm/opensm/osm_helper.c @@ -2,6 +2,7 @@ * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved. * Copyright (c) 1996-2003 Intel Corporation. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -1720,7 +1721,10 @@ osm_dump_notice(IN osm_log_t * const p_log, { if (osm_log_is_active(p_log, log_level)) { if (ib_notice_is_generic(p_ntci)) { + int i; char buff[1024]; + char line[BUF_SIZE]; + buff[0] = '\0'; /* immediate data based on the trap */ @@ -1758,11 +1762,16 @@ osm_dump_notice(IN osm_log_t * const p_log, case 144: sprintf(buff, "\t\t\t\tlid......................%u\n" - "\t\t\t\tnew_cap_mask.............0x%08x\n", + "\t\t\t\tlocal_changes............%d\n" + "\t\t\t\tnew_cap_mask.............0x%08x\n" + "\t\t\t\tchange_flags.............0x%x\n", cl_ntoh16(p_ntci->data_details.ntc_144. lid), + p_ntci->data_details.ntc_144.local_changes, cl_ntoh32(p_ntci->data_details.ntc_144. - new_cap_mask)); + new_cap_mask), + cl_ntoh16(p_ntci->data_details.ntc_144. + change_flgs)); break; case 145: sprintf(buff, @@ -1774,6 +1783,95 @@ osm_dump_notice(IN osm_log_t * const p_log, cl_ntoh64(p_ntci->data_details.ntc_145. new_sys_guid)); break; + case 256: + sprintf(buff, + "\t\t\t\tlid......................%u\n" + "\t\t\t\tdrslid...................%u\n" + "\t\t\t\tmethod...................0x%x\n" + "\t\t\t\tattr_id..................0x%x\n" + "\t\t\t\tattr_mod.................0x%x\n" + "\t\t\t\tm_key....................0x%016" PRIx64 "\n" + "\t\t\t\tdr_notice................%d\n" + "\t\t\t\tdr_path_truncated........%d\n" + "\t\t\t\tdr_hop_count.............%u\n", + cl_ntoh16(p_ntci->data_details.ntc_256.lid), + cl_ntoh16(p_ntci->data_details.ntc_256.dr_slid), + p_ntci->data_details.ntc_256.method, + cl_ntoh16(p_ntci->data_details.ntc_256.attr_id), + cl_ntoh32(p_ntci->data_details.ntc_256.attr_mod), + cl_ntoh64(p_ntci->data_details.ntc_256.mkey), + p_ntci->data_details.ntc_256.dr_trunc_hop >> 7, + p_ntci->data_details.ntc_256.dr_trunc_hop >> 6, + p_ntci->data_details.ntc_256.dr_trunc_hop & 0x3f); + sprintf(line, "Directed Path Dump of %u hop path:" + "\n\t\t\t\tPath = ", + p_ntci->data_details.ntc_256.dr_trunc_hop & 0x3f); + strcat(buff, line); + for (i = 0; + i <= (p_ntci->data_details.ntc_256.dr_trunc_hop & 0x3f); + i++) { + if (i == 0) + sprintf(line, "%d", + p_ntci->data_details.ntc_256.dr_rtn_path[i]); + else + sprintf(line, ",%d", + p_ntci->data_details.ntc_256.dr_rtn_path[i]); + strcat(buff, line); + } + break; + case 257: + case 258: + sprintf(buff, + "\t\t\t\tlid1.....................%u\n" + "\t\t\t\tlid2.....................%u\n" + "\t\t\t\tkey......................0x%x\n" + "\t\t\t\tsl.......................%d\n" + "\t\t\t\tqp1......................0x%x\n" + "\t\t\t\tqp2......................0x%x\n" + "\t\t\t\tgid1.....................0x%016" PRIx64 " : " + "0x%016" PRIx64 "\n" + "\t\t\t\tgid2.....................0x%016" PRIx64 " : " + "0x%016" PRIx64 "\n", + cl_ntoh16(p_ntci->data_details.ntc_257_258.lid1), + cl_ntoh16(p_ntci->data_details.ntc_257_258.lid2), + cl_ntoh32(p_ntci->data_details.ntc_257_258.key), + cl_ntoh32(p_ntci->data_details.ntc_257_258.qp1) >> 24, + cl_ntoh32(p_ntci->data_details.ntc_257_258.qp1) & 0xffffff, + cl_ntoh32(p_ntci->data_details.ntc_257_258.qp2), + cl_ntoh64(p_ntci->data_details.ntc_257_258.gid1.unicast.prefix), + cl_ntoh64(p_ntci->data_details.ntc_257_258.gid1.unicast.interface_id), + cl_ntoh64(p_ntci->data_details.ntc_257_258.gid2.unicast.prefix), + cl_ntoh64(p_ntci->data_details.ntc_257_258.gid2.unicast.interface_id)); + break; + case 259: + sprintf(buff, + "\t\t\t\tdata_valid...............0x%x\n" + "\t\t\t\tlid1.....................%u\n" + "\t\t\t\tlid2.....................%u\n" + "\t\t\t\tpkey.....................0x%x\n" + "\t\t\t\tsl.......................%d\n" + "\t\t\t\tqp1......................0x%x\n" + "\t\t\t\tqp2......................0x%x\n" + "\t\t\t\tgid1.....................0x%016" PRIx64 " : " + "0x%016" PRIx64 "\n" + "\t\t\t\tgid2.....................0x%016" PRIx64 " : " + "0x%016" PRIx64 "\n" + "\t\t\t\tsw_lid...................%u\n" + "\t\t\t\tport_no..................%u\n", + cl_ntoh16(p_ntci->data_details.ntc_259.data_valid), + cl_ntoh16(p_ntci->data_details.ntc_259.lid1), + cl_ntoh16(p_ntci->data_details.ntc_259.lid2), + cl_ntoh16(p_ntci->data_details.ntc_259.pkey), + cl_ntoh32(p_ntci->data_details.ntc_259.sl_qp1) >> 24, + cl_ntoh32(p_ntci->data_details.ntc_259.sl_qp1) & 0xffffff, + cl_ntoh32(p_ntci->data_details.ntc_259.qp2), + cl_ntoh64(p_ntci->data_details.ntc_259.gid1.unicast.prefix), + cl_ntoh64(p_ntci->data_details.ntc_259.gid1.unicast.interface_id), + cl_ntoh64(p_ntci->data_details.ntc_259.gid2.unicast.prefix), + cl_ntoh64(p_ntci->data_details.ntc_259.gid2.unicast.interface_id), + cl_ntoh16(p_ntci->data_details.ntc_259.sw_lid), + p_ntci->data_details.ntc_259.port_no); + break; } osm_log(p_log, log_level, From sashak at voltaire.com Wed Mar 18 05:10:59 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 18 Mar 2009 14:10:59 +0200 Subject: [ofa-general] Re: [PATCH] ib_types.h: Adding BKEY violation trap (259) In-Reply-To: <200903171614.41707.itaib@mellanox.com> References: <200903171614.41707.itaib@mellanox.com> Message-ID: <20090318121048.GG604@sashak.voltaire.com> On 16:14 Tue 17 Mar , Itai Baz wrote: > Hi Sasha, > > The following patch adds B_Key violation trap definition. > > Signed-off-by: Itai Baz Applied. Thanks. Sasha From sashak at voltaire.com Wed Mar 18 05:11:53 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 18 Mar 2009 14:11:53 +0200 Subject: [ofa-general] Re: [PATCH] libibmad/serv.c: Fixed respond function to return proper result code In-Reply-To: <200903171617.56184.itaib@mellanox.com> References: <200903171617.56184.itaib@mellanox.com> Message-ID: <20090318121153.GH604@sashak.voltaire.com> On 16:17 Tue 17 Mar , Itai Baz wrote: > Hi Sasha, > > mad_respond_via() function was always returning OK. > Fixing it to return the proper status. > > Signed-off-by: Itai Baz Applied. Thanks. Sasha From jrlang at uwyo.edu Wed Mar 18 07:37:36 2009 From: jrlang at uwyo.edu (jeffrey Lang) Date: Wed, 18 Mar 2009 08:37:36 -0600 Subject: [ofa-general] Need help with Infiniband problem In-Reply-To: References: <49BFBA94.5000709@uwyo.edu> <49C01E5B.5070703@uwyo.edu> Message-ID: <49C10730.4050003@uwyo.edu> Here's the output for ibchecknet: [root at h2o01 ~]# ibchecknet perfquery: iberror: failed: smp query nodeinfo: Node type not CA Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port all: FAILED #warn: counter SymbolErrors = 43259 (threshold 10) lid 1 port 17 Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port 17: FAILED #warn: counter LinkRecovers = 207 (threshold 10) lid 1 port 2 #warn: counter RcvErrors = 112 (threshold 10) lid 1 port 2 Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port 2: FAILED #warn: counter LinkDowned = 10 (threshold 10) lid 1 port 1 #warn: counter RcvErrors = 95 (threshold 10) lid 1 port 1 Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port 1: FAILED # Checking Ca: nodeguid 0x00066a0098007e99 # Checking Ca: nodeguid 0x00066a0098007e9b # Checking Ca: nodeguid 0x00066a0098007e97 # Checking Ca: nodeguid 0x00066a0098007e8c # Checking Ca: nodeguid 0x00066a0098007e94 # Checking Ca: nodeguid 0x00066a0098007e93 # Checking Ca: nodeguid 0x00066a0098007e8e # Checking Ca: nodeguid 0x00066a0098007e90 # Checking Ca: nodeguid 0x00066a0098007e98 # Checking Ca: nodeguid 0x00066a0098007e95 # Checking Ca: nodeguid 0x00066a0098007e8f # Checking Ca: nodeguid 0x00066a0098007e92 # Checking Ca: nodeguid 0x00066a0098007e8d # Checking Ca: nodeguid 0x00066a0098007e91 # Checking Ca: nodeguid 0x00066a0098007e96 # Checking Ca: nodeguid 0x00066a0098007e9c ## Summary: 17 nodes checked, 0 bad nodes found ## 32 ports checked, 0 bad ports found ## 3 ports have errors beyond threshold Silverstorm port display I see these messages in the switch log now: E|2009/03/18 07:34:28.635S: Thread "esm_sar" (0x83394a90) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o12 HCA-1, Port 0x00066A00A0007E8E, LID 0x000C, returning status 0x0600 : 0 I would have to assume that this is my problem, but how to fix? jeff Hal Rosenstock wrote: > On Tue, Mar 17, 2009 at 6:04 PM, jeffrey Lang wrote: > >> Here's the output smpquery portinfo -D 0 as requested below: >> >> [root at h2o01 ~]# smpquery portinfo -D 0 >> # Port info: DR path 0 port 0 >> Mkey:............................0x0000000000000000 >> GidPrefix:.......................0xfe80000000000000 >> Lid:.............................0x0003 >> SMLid:...........................0x0001 >> CapMask:.........................0x2510a68 >> IsTrapSupported >> IsAutomaticMigrationSupported >> IsSLMappingSupported >> IsLedInfoSupported >> IsSystemImageGUIDsupported >> IsCommunicatonManagementSupported >> IsVendorClassSupported >> IsCapabilityMaskNoticeSupported >> IsClientRegistrationSupported >> DiagCode:........................0x0000 >> MkeyLeasePeriod:.................0 >> LocalPort:.......................1 >> LinkWidthEnabled:................1X or 4X >> LinkWidthSupported:..............1X or 4X >> LinkWidthActive:.................4X >> LinkSpeedSupported:..............2.5 Gbps >> LinkState:.......................Active >> PhysLinkState:...................LinkUp >> LinkDownDefState:................Polling >> ProtectBits:.....................0 >> LMC:.............................0 >> LinkSpeedActive:.................2.5 Gbps >> LinkSpeedEnabled:................2.5 Gbps >> NeighborMTU:.....................2048 >> SMSL:............................0 >> VLCap:...........................VL0-3 >> InitType:........................0x00 >> VLHighLimit:.....................0 >> VLArbHighCap:....................8 >> VLArbLowCap:.....................8 >> InitReply:.......................0x00 >> MtuCap:..........................2048 >> VLStallCount:....................7 >> HoqLife:.........................0 >> OperVLs:.........................VL0-3 >> PartEnforceInb:..................0 >> PartEnforceOutb:.................0 >> FilterRawInb:....................0 >> FilterRawOutb:...................0 >> MkeyViolations:..................0 >> PkeyViolations:..................0 >> QkeyViolations:..................0 >> GuidCap:.........................32 >> ClientReregister:................0 >> SubnetTimeout:...................17 >> RespTimeVal:.....................16 >> LocalPhysErr:....................15 >> OverrunErr:......................15 >> MaxCreditHint:...................0 >> RoundTrip:.......................0 >> > > Looks fine. > > >> I did some checking, and It's not just this node having problems, all nodes >> seem to be having this same problem. >> > > Would you also run ibchecknet ? > > What error messages are on the SM side ? > > -- Hal > > >> jeff >> >> >> Hal Rosenstock wrote: >> >> 2009/3/17 jeffrey Lang : >> >> >> First let me say, I hope this is the right list for this email, if not >> please forgive me. >> I have a small 16 node compute cluster. The university where I work at >> recently opened a new Datacenter. My cluster was moved from the old >> Datacenter. Before the move the inifiniband was working properly, after >> the move the ipoib has stopped working. >> The cluster runs Centos 4 with all the latest updates and the Centos >> distributed OFED code. My plan was to update the OFED code once things had >> restablized. >> For the move, I shutdown the cluster, removed the inifiniband cables and the >> cluster was moved. I then reinstalled the infiniband cables (not in the >> same order before the move) and brought every thing back up. >> When i brought the cluster back up the ipoib would not work. The only >> message in the log file is "Mar 15 04:04:32 h2o01 kernel: ib0: multicast >> join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22". >> >> >> I think that there may be a rate issue in terms of this node relative >> to the IPoIB broadcast group which by default is 10 Gbps (4x SDR). >> What does this node's portinfo show (smpquery portinfo -D 0) in terms >> of link width and speed ? >> -- Hal >> >> >> The master node can see all the systems: >> [root at h2o01 log]# ibnodes >> Ca : 0x00066a0098007e99 ports 1 "h2o17 HCA-1" >> Ca : 0x00066a0098007e9b ports 1 "h2o18 HCA-1" >> Ca : 0x00066a0098007e97 ports 1 "h2o16 HCA-1" >> Ca : 0x00066a0098007e8c ports 1 "h2o15 HCA-1" >> Ca : 0x00066a0098007e94 ports 1 "h2o14 HCA-1" >> Ca : 0x00066a0098007e93 ports 1 "h2o13 HCA-1" >> Ca : 0x00066a0098007e8e ports 1 "h2o12 HCA-1" >> Ca : 0x00066a0098007e90 ports 1 "h2o11 HCA-1" >> Ca : 0x00066a0098007e98 ports 1 "h2o10 HCA-1" >> Ca : 0x00066a0098007e95 ports 1 "h2o09 HCA-1" >> Ca : 0x00066a0098007e8f ports 1 "h2o08 HCA-1" >> Ca : 0x00066a0098007e92 ports 1 "h2o07 HCA-1" >> Ca : 0x00066a0098007e8d ports 1 "h2o06 HCA-1" >> Ca : 0x00066a0098007e91 ports 1 "h2o05 HCA-1" >> Ca : 0x00066a0098007e96 ports 1 "h2ocfs HCA-1" >> Ca : 0x00066a0098007e9c ports 1 "h2o01 HCA-1" >> Switch : 0x00066a00d8000593 ports 24 "SilverStorm 9024 >> GUID=0x00066a00d8000593" enhanced port 0 lid 1 lmc 0 >> I've reset the sm on the switch, but nothing seems to work. >> Any ideas of where to look for whats causing the problem? >> jeff >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: silverstorm Type: image/png Size: 235954 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: jrlang.vcf Type: text/x-vcard Size: 311 bytes Desc: not available URL: From jrlang at uwyo.edu Wed Mar 18 07:45:30 2009 From: jrlang at uwyo.edu (jeffrey Lang) Date: Wed, 18 Mar 2009 08:45:30 -0600 Subject: ***SPAM*** Re: [ofa-general] Need help with Infiniband problem In-Reply-To: References: <49BFBA94.5000709@uwyo.edu> <49C01E5B.5070703@uwyo.edu> Message-ID: <49C1090A.2070007@uwyo.edu> I did a soft reset on the switch to see what errors would appear during the reboot. Below is what was in the log after the reboot: W|2009/03/18 07:36:41.998S: Thread "ssiSpawnDelayedResetCardTask" (0x831d8530) Osa: Info, rebooting the local card. W|2009/03/18 07:36:42.928S: Thread "tChModWkr101" (0x8321acd0) TMS: CMS Warning: Info, slot 101 reboot event has been received! ================================================================================ I|2009/03/18 14:37:08.010U: Thread "startup" (0x87dbf6c0) Log: Recovered 9621 bytes of log data from persistent store I|2009/03/18 14:37:08.010U: Thread "startup" (0x87dbf6c0) Log: Recovered 9621 bytes of log data from persistent store A|2009/03/18 14:37:16.980U: Thread "startup" (0x87dbf6c0) boot: I9K t3 Firmware Booting Version: 3.4.0.11.3 A|2009/03/18 14:37:16.980U: Thread "startup" (0x87dbf6c0) boot: Memory: Physical: 0x8000000 Available: 0x8000000 Reserved: 0x0 A|2009/03/18 14:37:16.980U: Thread "startup" (0x87dbf6c0) boot: Reboot cause (9): Operator requested. E|2009/03/18 07:37:26.950S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2ocfs HCA-1, Port 0x00066A00A0007E96, LID 0x0002, returning status 0x0600 : 0 E|2009/03/18 07:37:27.050S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o08 HCA-1, Port 0x00066A00A0007E8F, LID 0x0009, returning status 0x0600 : 0 E|2009/03/18 07:37:27.130S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o05 HCA-1, Port 0x00066A00A0007E91, LID 0x0006, returning status 0x0600 : 0 E|2009/03/18 07:37:27.130S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o01 HCA-1, Port 0x00066A00A0007E9C, LID 0x0003, returning status 0x0600 : 0 E|2009/03/18 07:37:27.130S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o06 HCA-1, Port 0x00066A00A0007E8D, LID 0x0007, returning status 0x0600 : 0 E|2009/03/18 07:37:27.130S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o07 HCA-1, Port 0x00066A00A0007E92, LID 0x0008, returning status 0x0600 : 0 E|2009/03/18 07:37:27.130S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o09 HCA-1, Port 0x00066A00A0007E95, LID 0x0013, returning status 0x0600 : 0 E|2009/03/18 07:37:27.130S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o10 HCA-1, Port 0x00066A00A0007E98, LID 0x000A, returning status 0x0600 : 0 E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o12 HCA-1, Port 0x00066A00A0007E8E, LID 0x000C, returning status 0x0600 : 0 E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o13 HCA-1, Port 0x00066A00A0007E93, LID 0x000D, returning status 0x0600 : 0 E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o11 HCA-1, Port 0x00066A00A0007E90, LID 0x000B, returning status 0x0600 : 0 E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o15 HCA-1, Port 0x00066A00A0007E8C, LID 0x000E, returning status 0x0600 : 0 E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o14 HCA-1, Port 0x00066A00A0007E94, LID 0x0010, returning status 0x0600 : 0 E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: (No Memory) : 0 E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: (No Memory) : 0 E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: (No Memory) : 0 E|2009/03/18 07:37:29.130S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o05 HCA-1, Port 0x00066A00A0007E91, LID 0x0006, returning status 0x0600 : 0 E|2009/03/18 07:37:29.130S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o01 HCA-1, Port 0x00066A00A0007E9C, LID 0x0003, returning status 0x0600 : 0 E|2009/03/18 07:37:29.130S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o06 HCA-1, Port 0x00066A00A0007E8D, LID 0x0007, returning status 0x0600 : 0 E|2009/03/18 07:37:29.130S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o09 HCA-1, Port 0x00066A00A0007E95, LID 0x0013, returning status 0x0600 : 0 E|2009/03/18 07:37:29.130S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o07 HCA-1, Port 0x00066A00A0007E92, LID 0x0008, returning status 0x0600 : 0 E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o10 HCA-1, Port 0x00066A00A0007E98, LID 0x000A, returning status 0x0600 : 0 E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o13 HCA-1, Port 0x00066A00A0007E93, LID 0x000D, returning status 0x0600 : 0 E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o11 HCA-1, Port 0x00066A00A0007E90, LID 0x000B, returning status 0x0600 : 0 E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o12 HCA-1, Port 0x00066A00A0007E8E, LID 0x000C, returning status 0x0600 : 0 E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o16 HCA-1, Port 0x00066A00A0007E97, LID 0x000F, returning status 0x0600 : 0 E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o15 HCA-1, Port 0x00066A00A0007E8C, LID 0x000E, returning status 0x0600 : 0 E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: (No Memory) : 0 E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: (No Memory) : 0 E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: (No Memory) : 0 E|2009/03/18 07:37:32.283S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2ocfs HCA-1, Port 0x00066A00A0007E96, LID 0x0002, returning status 0x0600 : 0 E|2009/03/18 07:37:32.383S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o08 HCA-1, Port 0x00066A00A0007E8F, LID 0x0009, returning status 0x0600 : 0 E|2009/03/18 07:37:34.463S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o05 HCA-1, Port 0x00066A00A0007E91, LID 0x0006, returning status 0x0600 : 0 E|2009/03/18 07:37:34.463S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o01 HCA-1, Port 0x00066A00A0007E9C, LID 0x0003, returning status 0x0600 : 0 E|2009/03/18 07:37:34.463S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o09 HCA-1, Port 0x00066A00A0007E95, LID 0x0013, returning status 0x0600 : 0 E|2009/03/18 07:37:34.463S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o06 HCA-1, Port 0x00066A00A0007E8D, LID 0x0007, returning status 0x0600 : 0 E|2009/03/18 07:37:34.463S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o07 HCA-1, Port 0x00066A00A0007E92, LID 0x0008, returning status 0x0600 : 0 E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o10 HCA-1, Port 0x00066A00A0007E98, LID 0x000A, returning status 0x0600 : 0 E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o12 HCA-1, Port 0x00066A00A0007E8E, LID 0x000C, returning status 0x0600 : 0 E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o13 HCA-1, Port 0x00066A00A0007E93, LID 0x000D, returning status 0x0600 : 0 E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o11 HCA-1, Port 0x00066A00A0007E90, LID 0x000B, returning status 0x0600 : 0 E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o16 HCA-1, Port 0x00066A00A0007E97, LID 0x000F, returning status 0x0600 : 0 E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o15 HCA-1, Port 0x00066A00A0007E8C, LID 0x000E, returning status 0x0600 : 0 E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: (No Memory) : 0 E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: (No Memory) : 0 E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: (No Memory) : 0 E|2009/03/18 07:37:40.283S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2ocfs HCA-1, Port 0x00066A00A0007E96, LID 0x0002, returning status 0x0600 : 0 E|2009/03/18 07:37:40.383S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o08 HCA-1, Port 0x00066A00A0007E8F, LID 0x0009, returning status 0x0600 : 0 E|2009/03/18 07:37:42.463S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o05 HCA-1, Port 0x00066A00A0007E91, LID 0x0006, returning status 0x0600 : 0 E|2009/03/18 07:37:42.463S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o01 HCA-1, Port 0x00066A00A0007E9C, LID 0x0003, returning status 0x0600 : 0 E|2009/03/18 07:37:42.463S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o09 HCA-1, Port 0x00066A00A0007E95, LID 0x0013, returning status 0x0600 : 0 E|2009/03/18 07:37:42.463S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o06 HCA-1, Port 0x00066A00A0007E8D, LID 0x0007, returning status 0x0600 : 0 E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o07 HCA-1, Port 0x00066A00A0007E92, LID 0x0008, returning status 0x0600 : 0 E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o10 HCA-1, Port 0x00066A00A0007E98, LID 0x000A, returning status 0x0600 : 0 E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o12 HCA-1, Port 0x00066A00A0007E8E, LID 0x000C, returning status 0x0600 : 0 E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o13 HCA-1, Port 0x00066A00A0007E93, LID 0x000D, returning status 0x0600 : 0 E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o11 HCA-1, Port 0x00066A00A0007E90, LID 0x000B, returning status 0x0600 : 0 E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o15 HCA-1, Port 0x00066A00A0007E8C, LID 0x000E, returning status 0x0600 : 0 E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o16 HCA-1, Port 0x00066A00A0007E97, LID 0x000F, returning status 0x0600 : 0 E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: (No Memory) : 0 E|2009/03/18 07:37:42.483S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: (No Memory) : 0 E|2009/03/18 07:37:42.483S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: (No Memory) : 0 E|2009/03/18 07:37:56.273S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2ocfs HCA-1, Port 0x00066A00A0007E96, LID 0x0002, returning status 0x0600 : 0 E|2009/03/18 07:37:56.383S: Thread "esm_sar" (0x832c36a0) ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of 0x0000000000010083 does not have bits required to create a group (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for request from h2o08 HCA-1, Port 0x00066A00A0007E8F, LID 0x0009, returning status 0x0600 : 0 Here's the output of the saquery, this was done after the soft reboot: [root at h2o01 ~]# saquery -g saquery: iba_init failed (status=0x1): FERROR Does this mean my switch has gone and died? jeff Hal Rosenstock wrote: > On Tue, Mar 17, 2009 at 6:22 PM, Hal Rosenstock > wrote: > >> Would you also run ibchecknet ? >> >> What error messages are on the SM side ? >> > > What does saquery -g show ? > > You did say this was a switch based SM, right ? > > -- Hal > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: jrlang.vcf Type: text/x-vcard Size: 311 bytes Desc: not available URL: From jrlang at uwyo.edu Wed Mar 18 07:52:42 2009 From: jrlang at uwyo.edu (jeffrey Lang) Date: Wed, 18 Mar 2009 08:52:42 -0600 Subject: [ofa-general] Any documents Message-ID: <49C10ABA.4010709@uwyo.edu> Being new and not totally understanding of the in's and out's of infiniband, are there any documents, tutorials, etc that would help me get a better working understanding of how infiniband and it's software is all put together? jeff -------------- next part -------------- A non-text attachment was scrubbed... Name: jrlang.vcf Type: text/x-vcard Size: 298 bytes Desc: not available URL: From rdreier at cisco.com Wed Mar 18 08:09:41 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Mar 2009 08:09:41 -0700 Subject: [ofa-general] possible bugs in IB code usage of min() and max() In-Reply-To: (Or Gerlitz's message of "Wed, 18 Mar 2009 10:18:36 +0200 (IST)") References: Message-ID: > I came across these sparse warnings, are they false-negative or bugs > that should be fixed? Did you look at the code in question? What's your opinion? From sean.hefty at intel.com Wed Mar 18 08:14:26 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 18 Mar 2009 08:14:26 -0700 Subject: [ofa-general] RE: [PATCH] add c99 definitions within the ib_mad_f structure In-Reply-To: <49C0C64B.6050606@gmail.com> References: <49B4CFC6.7020800@gmail.com> <20090309154000.GL15398@sashak.voltaire.com> <8436BA8017D247AC9287E3F0DB4458DB@amr.corp.intel.com> <20090309113233.b382dc70.weiny2@llnl.gov> <20090309113530.8b056000.weiny2@llnl.gov> <49C0C64B.6050606@gmail.com> Message-ID: <2928B0C2CC3443D7AF84C19200BFBC87@amr.corp.intel.com> this knowingly breaks the windows build... From swise at opengridcomputing.com Wed Mar 18 08:48:03 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 18 Mar 2009 10:48:03 -0500 Subject: [ofa-general] [PATCH v3] RDMA/cxgb3: Handle EEH events for active connections. In-Reply-To: <20090309102014.5738.98445.stgit@build.ogc.int> References: <20090309102014.5738.98445.stgit@build.ogc.int> Message-ID: <49C117B3.7050306@opengridcomputing.com> Hey Roland, Have you had a chance to review this and the other patch I sent out for inclusion in 2.6.30? Thanks! Steve. From dorfman.eli at gmail.com Wed Mar 18 09:34:54 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Wed, 18 Mar 2009 18:34:54 +0200 Subject: [ofa-general] Re: [PATCH] add c99 definitions within the ib_mad_f structure In-Reply-To: <2928B0C2CC3443D7AF84C19200BFBC87@amr.corp.intel.com> References: <49B4CFC6.7020800@gmail.com> <20090309154000.GL15398@sashak.voltaire.com> <8436BA8017D247AC9287E3F0DB4458DB@amr.corp.intel.com> <20090309113233.b382dc70.weiny2@llnl.gov> <20090309113530.8b056000.weiny2@llnl.gov> <49C0C64B.6050606@gmail.com> <2928B0C2CC3443D7AF84C19200BFBC87@amr.corp.intel.com> Message-ID: <49C122AE.70506@gmail.com> Sean Hefty wrote: > this knowingly breaks the windows build... > So what do you suggest? Ira replied that he has no problems with it. From arlin.r.davis at intel.com Wed Mar 18 10:01:30 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Wed, 18 Mar 2009 10:01:30 -0700 Subject: [ofa-general] RE: [PATCH] add c99 definitions within the ib_mad_f structure In-Reply-To: <49C122AE.70506@gmail.com> References: <49B4CFC6.7020800@gmail.com> <20090309154000.GL15398@sashak.voltaire.com> <8436BA8017D247AC9287E3F0DB4458DB@amr.corp.intel.com> <20090309113233.b382dc70.weiny2@llnl.gov> <20090309113530.8b056000.weiny2@llnl.gov> <49C0C64B.6050606@gmail.com> <2928B0C2CC3443D7AF84C19200BFBC87@amr.corp.intel.com> <49C122AE.70506@gmail.com> Message-ID: >Sean Hefty wrote: >> this knowingly breaks the windows build... >> >So what do you suggest? >Ira replied that he has no problems with it. > What happened? I thought we were all in agreement when we advocated removal of c99 definitions. I think this is a small price to pay for mantainability and portability across OS'es. Look at the state of OpenSM. The Windows version is so outdated that it is virtually unusable and is very limited in feature's as compared to OFED versions. Is this what we want for these components? From weiny2 at llnl.gov Wed Mar 18 10:24:17 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Wed, 18 Mar 2009 10:24:17 -0700 Subject: [ofa-general] Re: [PATCH] add c99 definitions within the ib_mad_f structure In-Reply-To: <49C122AE.70506@gmail.com> References: <49B4CFC6.7020800@gmail.com> <20090309154000.GL15398@sashak.voltaire.com> <8436BA8017D247AC9287E3F0DB4458DB@amr.corp.intel.com> <20090309113233.b382dc70.weiny2@llnl.gov> <20090309113530.8b056000.weiny2@llnl.gov> <49C0C64B.6050606@gmail.com> <2928B0C2CC3443D7AF84C19200BFBC87@amr.corp.intel.com> <49C122AE.70506@gmail.com> Message-ID: <20090318102417.8eb06001.weiny2@llnl.gov> On Wed, 18 Mar 2009 18:34:54 +0200 "Eli Dorfman (Voltaire)" wrote: > Sean Hefty wrote: > > this knowingly breaks the windows build... > > > So what do you suggest? > Ira replied that he has no problems with it. I am sorry if I was unclear but I was speaking in _favor_ of having Windows compatibility. I meant I have no problem maintaining windows compatibility but was mentioning the fact that I don't have a way to test this (and I am sure Sean knows that many other developers here are in the same situation). Therefore, I was hoping that I would be forgiven if I accidentally broke the build. I too am a bit frustrated by this patch. Mainly because Sasha just accepted the exact opposite patch about a month ago! Ira For reference the conversation was: > On Mon, 9 Mar 2009 11:15:17 -0700 > "Sean Hefty" wrote: > > > > This is getting old and frustrating. Do the OFA IB vendors not want to support > > Windows based systems connected to their hardware? > > > I have no problem with it but I don't have a Windows machine to test on. So this will make the effort harder on your part Sean, sorry. Ira -- Ira Weiny Math Programer/Computer Scientist Larence Livermore National Lab weiny2 at llnl.gov From sean.hefty at intel.com Wed Mar 18 10:51:14 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 18 Mar 2009 10:51:14 -0700 Subject: [ofa-general] RE: [PATCH] add c99 definitions within the ib_mad_f structure In-Reply-To: <49C122AE.70506@gmail.com> References: <49B4CFC6.7020800@gmail.com> <20090309154000.GL15398@sashak.voltaire.com> <8436BA8017D247AC9287E3F0DB4458DB@amr.corp.intel.com> <20090309113233.b382dc70.weiny2@llnl.gov> <20090309113530.8b056000.weiny2@llnl.gov> <49C0C64B.6050606@gmail.com> <2928B0C2CC3443D7AF84C19200BFBC87@amr.corp.intel.com> <49C122AE.70506@gmail.com> Message-ID: <3A365155DA7D48D58C819AC907C03FA2@amr.corp.intel.com> >So what do you suggest? Changing the WinOF build environment is something that could be brought up in Sonoma, if there will be enough representatives there. Alternatively, WinOF schedules regular con-calls. >Ira replied that he has no problems with it. I remember Ira stating that he couldn't build or test his patches on Windows. I have no problem with that. I don't pull the ib-mgmt.git tree every day. When I do pull, if I hit into any build issues, I'll just correct them and submit a patch. - Sean From sashak at voltaire.com Wed Mar 18 13:26:39 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Wed, 18 Mar 2009 22:26:39 +0200 Subject: [ofa-general] Re: [PATCH] libibmad: added support for handling of BM (Baseboard management) MADs In-Reply-To: <200903171622.34568.itaib@mellanox.com> References: <200903171622.34568.itaib@mellanox.com> Message-ID: <20090318202638.GA20159@sashak.voltaire.com> Hi Itai, On 16:22 Tue 17 Mar , Itai Baz wrote: > diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h > index b8290a7..efc887a 100644 > --- a/libibmad/include/infiniband/mad.h > +++ b/libibmad/include/infiniband/mad.h > @@ -876,6 +922,9 @@ MAD_EXPORT uint8_t *performance_reset_via(void *rcvbuf, ib_portid_t * dest, > unsigned timeout, unsigned id, > const struct ibmad_port *srcport); > > +/* bm.c */ > +uint8_t * bm_rpc_call_via(void *data, ib_portid_t *portid, ib_bm_call_t *call, struct ibmad_port *srcport); > + I think that MAD_EXPORT is needed here in order to not break WinOF compatibility. > /* dump.c */ > MAD_EXPORT ib_mad_dump_fn > mad_dump_int, mad_dump_uint, mad_dump_hex, mad_dump_rhex, > diff --git a/libibmad/src/bm.c b/libibmad/src/bm.c > new file mode 100644 > index 0000000..a095b9a > --- /dev/null > +++ b/libibmad/src/bm.c > @@ -0,0 +1,101 @@ > +/* > + * Copyright (c) 2004-2007 Voltaire Inc. All rights reserved. > + * > + * This software is available to you under a choice of one of two > + * licenses. You may choose to be licensed under the terms of the GNU > + * General Public License (GPL) Version 2, available from the file > + * COPYING in the main directory of this source tree, or the > + * OpenIB.org BSD license below: > + * > + * Redistribution and use in source and binary forms, with or > + * without modification, are permitted provided that the following > + * conditions are met: > + * > + * - Redistributions of source code must retain the above > + * copyright notice, this list of conditions and the following > + * disclaimer. > + * > + * - Redistributions in binary form must reproduce the above > + * copyright notice, this list of conditions and the following > + * disclaimer in the documentation and/or other materials > + * provided with the distribution. > + * > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, > + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF > + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND > + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS > + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN > + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN > + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE > + * SOFTWARE. > + * > + */ > + > +#if HAVE_CONFIG_H > +# include > +#endif /* HAVE_CONFIG_H */ > + > +#include > +#include > +#include > +#include > +#include > +#include > + > +#include > + > +#undef DEBUG > +#define DEBUG if (ibdebug) IBWARN > + > +static inline int > +response_expected(int method) > +{ > + return method == IB_MAD_METHOD_GET || > + method == IB_MAD_METHOD_SET || > + method == IB_MAD_METHOD_TRAP; > +} > + > +uint8_t * > +bm_rpc_call_via(void *data, ib_portid_t *portid, ib_bm_call_t *call, struct ibmad_port *srcport) This function uses rpc or regular sends. Maybe slightly better name would be just 'bm_call_via'? > +{ > + ib_rpc_t rpc = {0}; > + int resp_expected; > + char data_with_bkey[IB_BM_BKEY_AND_DATA_SZ] = {0}; > + > + DEBUG("route %s data %p", portid2str(portid), data); > + if (portid->lid <= 0) { > + IBWARN("only lid routes are supported"); > + return 0; > + } > + > + resp_expected = response_expected(call->method); > + > + rpc.mgtclass = IB_BOARD_MGMT_CLASS; > + > + rpc.method = call->method; > + rpc.attr.id = call->attrid; > + rpc.attr.mod = call->mod; > + rpc.timeout = resp_expected ? call->timeout : 0; > + // send data and bkey > + rpc.datasz = IB_BM_BKEY_AND_DATA_SZ; > + rpc.dataoffs = IB_BM_BKEY_OFFS; > + > + // copy data to a buffer which also includes the bkey > + *((uint64_t *) data_with_bkey) = call->bkey; Actually this means that B_Key should be provided in call->bkey in network byte order. It is not like most others "call" helpers work. Wouldn't it be better to keep compatibility here? > + memcpy(data_with_bkey + IB_BM_DATA_OFFS - IB_BM_BKEY_OFFS, data, IB_BM_DATA_SZ); > + > + DEBUG("method 0x%x attr 0x%x mod 0x%x datasz %d off %d res_ex %d bkey 0x%08x%08x", > + rpc.method, rpc.attr.id, rpc.attr.mod, > + rpc.datasz, rpc.dataoffs, resp_expected, > + (int) (call->bkey >> 32), (int) call->bkey); > + > + portid->qp = 1; > + if (!portid->qkey) > + portid->qkey = IB_DEFAULT_QP1_QKEY; > + > + if (resp_expected) > + return mad_rpc_rmpp(srcport, &rpc, portid, 0, data_with_bkey); /* FIXME: no RMPP for now */ How is the response data returned to the caller? data_with_bkey is a local buffer, right? > + > + return mad_send_via(&rpc, portid, 0, data_with_bkey, srcport) < 0 ? 0 : data_with_bkey; /* FIXME: no RMPP for now */ > +} Sasha From hal.rosenstock at gmail.com Wed Mar 18 13:56:04 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 18 Mar 2009 16:56:04 -0400 Subject: ***SPAM*** Re: ***SPAM*** Re: [ofa-general] Need help with Infiniband problem In-Reply-To: <49C1090A.2070007@uwyo.edu> References: <49BFBA94.5000709@uwyo.edu> <49C01E5B.5070703@uwyo.edu> <49C1090A.2070007@uwyo.edu> Message-ID: On Wed, Mar 18, 2009 at 10:45 AM, jeffrey Lang wrote: > I did a soft reset on the switch to see what errors would appear during the > reboot.  Below is what was in the log after the reboot: > > W|2009/03/18 07:36:41.998S: Thread "ssiSpawnDelayedResetCardTask" > (0x831d8530) > Osa: Info, rebooting the local card. > > W|2009/03/18 07:36:42.928S: Thread "tChModWkr101" (0x8321acd0) > TMS: CMS Warning: Info, slot 101 reboot event has been received! > > ================================================================================ > > I|2009/03/18 14:37:08.010U: Thread "startup" (0x87dbf6c0) > Log: Recovered 9621 bytes of log data from persistent store > > I|2009/03/18 14:37:08.010U: Thread "startup" (0x87dbf6c0) > Log: Recovered 9621 bytes of log data from persistent store > > A|2009/03/18 14:37:16.980U: Thread "startup" (0x87dbf6c0) > boot: I9K t3 Firmware Booting Version: 3.4.0.11.3 > > A|2009/03/18 14:37:16.980U: Thread "startup" (0x87dbf6c0) > boot: Memory: Physical: 0x8000000 Available: 0x8000000 Reserved: 0x0 > > A|2009/03/18 14:37:16.980U: Thread "startup" (0x87dbf6c0) > boot: Reboot cause (9): Operator requested. > > E|2009/03/18 07:37:26.950S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2ocfs HCA-1, Port 0x00066A00A0007E96, LID 0x0002, returning > status 0x0600 : 0 The IPoIB broadcast group needs to somehow be precreated at the SM. You'll need to contact your SM vendor as to how that is done. -- Hal > E|2009/03/18 07:37:27.050S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o08 HCA-1, Port 0x00066A00A0007E8F, LID 0x0009, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:27.130S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o05 HCA-1, Port 0x00066A00A0007E91, LID 0x0006, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:27.130S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o01 HCA-1, Port 0x00066A00A0007E9C, LID 0x0003, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:27.130S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o06 HCA-1, Port 0x00066A00A0007E8D, LID 0x0007, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:27.130S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o07 HCA-1, Port 0x00066A00A0007E92, LID 0x0008, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:27.130S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o09 HCA-1, Port 0x00066A00A0007E95, LID 0x0013, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:27.130S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o10 HCA-1, Port 0x00066A00A0007E98, LID 0x000A, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o12 HCA-1, Port 0x00066A00A0007E8E, LID 0x000C, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o13 HCA-1, Port 0x00066A00A0007E93, LID 0x000D, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o11 HCA-1, Port 0x00066A00A0007E90, LID 0x000B, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o15 HCA-1, Port 0x00066A00A0007E8C, LID 0x000E, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o14 HCA-1, Port 0x00066A00A0007E94, LID 0x0010, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: (No Memory) : 0 > > E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: (No Memory) : 0 > > E|2009/03/18 07:37:27.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: (No Memory) : 0 > > E|2009/03/18 07:37:29.130S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o05 HCA-1, Port 0x00066A00A0007E91, LID 0x0006, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:29.130S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o01 HCA-1, Port 0x00066A00A0007E9C, LID 0x0003, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:29.130S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o06 HCA-1, Port 0x00066A00A0007E8D, LID 0x0007, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:29.130S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o09 HCA-1, Port 0x00066A00A0007E95, LID 0x0013, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:29.130S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o07 HCA-1, Port 0x00066A00A0007E92, LID 0x0008, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o10 HCA-1, Port 0x00066A00A0007E98, LID 0x000A, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o13 HCA-1, Port 0x00066A00A0007E93, LID 0x000D, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o11 HCA-1, Port 0x00066A00A0007E90, LID 0x000B, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o12 HCA-1, Port 0x00066A00A0007E8E, LID 0x000C, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o16 HCA-1, Port 0x00066A00A0007E97, LID 0x000F, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o15 HCA-1, Port 0x00066A00A0007E8C, LID 0x000E, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: (No Memory) : 0 > > E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: (No Memory) : 0 > > E|2009/03/18 07:37:29.140S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: (No Memory) : 0 > > E|2009/03/18 07:37:32.283S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2ocfs HCA-1, Port 0x00066A00A0007E96, LID 0x0002, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:32.383S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o08 HCA-1, Port 0x00066A00A0007E8F, LID 0x0009, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:34.463S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o05 HCA-1, Port 0x00066A00A0007E91, LID 0x0006, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:34.463S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o01 HCA-1, Port 0x00066A00A0007E9C, LID 0x0003, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:34.463S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o09 HCA-1, Port 0x00066A00A0007E95, LID 0x0013, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:34.463S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o06 HCA-1, Port 0x00066A00A0007E8D, LID 0x0007, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:34.463S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o07 HCA-1, Port 0x00066A00A0007E92, LID 0x0008, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o10 HCA-1, Port 0x00066A00A0007E98, LID 0x000A, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o12 HCA-1, Port 0x00066A00A0007E8E, LID 0x000C, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o13 HCA-1, Port 0x00066A00A0007E93, LID 0x000D, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o11 HCA-1, Port 0x00066A00A0007E90, LID 0x000B, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o16 HCA-1, Port 0x00066A00A0007E97, LID 0x000F, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o15 HCA-1, Port 0x00066A00A0007E8C, LID 0x000E, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: (No Memory) : 0 > > E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: (No Memory) : 0 > > E|2009/03/18 07:37:34.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: (No Memory) : 0 > > E|2009/03/18 07:37:40.283S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2ocfs HCA-1, Port 0x00066A00A0007E96, LID 0x0002, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:40.383S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o08 HCA-1, Port 0x00066A00A0007E8F, LID 0x0009, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:42.463S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o05 HCA-1, Port 0x00066A00A0007E91, LID 0x0006, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:42.463S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o01 HCA-1, Port 0x00066A00A0007E9C, LID 0x0003, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:42.463S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o09 HCA-1, Port 0x00066A00A0007E95, LID 0x0013, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:42.463S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o06 HCA-1, Port 0x00066A00A0007E8D, LID 0x0007, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o07 HCA-1, Port 0x00066A00A0007E92, LID 0x0008, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o10 HCA-1, Port 0x00066A00A0007E98, LID 0x000A, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o12 HCA-1, Port 0x00066A00A0007E8E, LID 0x000C, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o13 HCA-1, Port 0x00066A00A0007E93, LID 0x000D, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o11 HCA-1, Port 0x00066A00A0007E90, LID 0x000B, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o15 HCA-1, Port 0x00066A00A0007E8C, LID 0x000E, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o16 HCA-1, Port 0x00066A00A0007E97, LID 0x000F, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:42.473S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: (No Memory) : 0 > > E|2009/03/18 07:37:42.483S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: (No Memory) : 0 > > E|2009/03/18 07:37:42.483S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: (No Memory) : 0 > > E|2009/03/18 07:37:56.273S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2ocfs HCA-1, Port 0x00066A00A0007E96, LID 0x0002, returning > status 0x0600 : 0 > > E|2009/03/18 07:37:56.383S: Thread "esm_sar" (0x832c36a0) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o08 HCA-1, Port 0x00066A00A0007E8F, LID 0x0009, returning > status 0x0600 : 0 > > Here's the output of the saquery, this was done after the soft reboot: > > [root at h2o01 ~]# saquery -g > saquery: iba_init failed (status=0x1): FERROR > > Does this mean my switch has gone and died? > > jeff > > Hal Rosenstock wrote: > > On Tue, Mar 17, 2009 at 6:22 PM, Hal Rosenstock > wrote: > > > Would you also run ibchecknet ? > What error messages are on the SM side ? > > > What does saquery -g show ? > You did say this was a switch based SM, right ? > -- Hal > From hal.rosenstock at gmail.com Wed Mar 18 14:02:59 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 18 Mar 2009 17:02:59 -0400 Subject: ***SPAM*** Re: [ofa-general] Need help with Infiniband problem In-Reply-To: <49C10730.4050003@uwyo.edu> References: <49BFBA94.5000709@uwyo.edu> <49C01E5B.5070703@uwyo.edu> <49C10730.4050003@uwyo.edu> Message-ID: On Wed, Mar 18, 2009 at 10:37 AM, jeffrey Lang wrote: > Here's the output for ibchecknet: > > [root at h2o01 ~]# ibchecknet > perfquery: iberror: failed: smp query nodeinfo: Node type not CA What diags version is being used ? > Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port all: > FAILED > #warn: counter SymbolErrors = 43259     (threshold 10) lid 1 port 17 > Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port 17: > FAILED > #warn: counter LinkRecovers = 207       (threshold 10) lid 1 port 2 > #warn: counter RcvErrors = 112  (threshold 10) lid 1 port 2 > Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port 2: > FAILED > #warn: counter LinkDowned = 10  (threshold 10) lid 1 port 1 > #warn: counter RcvErrors = 95   (threshold 10) lid 1 port 1 > Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port 1: > FAILED Are the counts for these ports (1,2,17) changing ? You can look with perfquery 1 . This is a separate issue from the lack of an IPoIB broadcast group assuming these numbers are incrementing. -- Hal > # Checking Ca: nodeguid 0x00066a0098007e99 > > # Checking Ca: nodeguid 0x00066a0098007e9b > > # Checking Ca: nodeguid 0x00066a0098007e97 > > # Checking Ca: nodeguid 0x00066a0098007e8c > > # Checking Ca: nodeguid 0x00066a0098007e94 > > # Checking Ca: nodeguid 0x00066a0098007e93 > > # Checking Ca: nodeguid 0x00066a0098007e8e > > # Checking Ca: nodeguid 0x00066a0098007e90 > > # Checking Ca: nodeguid 0x00066a0098007e98 > > # Checking Ca: nodeguid 0x00066a0098007e95 > > # Checking Ca: nodeguid 0x00066a0098007e8f > > # Checking Ca: nodeguid 0x00066a0098007e92 > > # Checking Ca: nodeguid 0x00066a0098007e8d > > # Checking Ca: nodeguid 0x00066a0098007e91 > > # Checking Ca: nodeguid 0x00066a0098007e96 > > # Checking Ca: nodeguid 0x00066a0098007e9c > > ## Summary: 17 nodes checked, 0 bad nodes found > ##          32 ports checked, 0 bad ports found > ##          3 ports have errors beyond threshold > > > > > I see these messages in the switch log now: > > E|2009/03/18 07:34:28.635S: Thread "esm_sar" (0x83394a90) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o12 HCA-1, Port 0x00066A00A0007E8E, LID 0x000C, returning > status 0x0600 : 0 > > > I would have to assume that this is my problem, but how to fix? > > jeff > > > > > > > Hal Rosenstock wrote: > > On Tue, Mar 17, 2009 at 6:04 PM, jeffrey Lang wrote: > > > Here's the output smpquery portinfo -D 0 as requested below: > [root at h2o01 ~]# smpquery portinfo -D 0 > # Port info: DR path 0 port 0 > Mkey:............................0x0000000000000000 > GidPrefix:.......................0xfe80000000000000 > Lid:.............................0x0003 > SMLid:...........................0x0001 > CapMask:.........................0x2510a68 >                 IsTrapSupported >                 IsAutomaticMigrationSupported >                 IsSLMappingSupported >                 IsLedInfoSupported >                 IsSystemImageGUIDsupported >                 IsCommunicatonManagementSupported >                 IsVendorClassSupported >                 IsCapabilityMaskNoticeSupported >                 IsClientRegistrationSupported > DiagCode:........................0x0000 > MkeyLeasePeriod:.................0 > LocalPort:.......................1 > LinkWidthEnabled:................1X or 4X > LinkWidthSupported:..............1X or 4X > LinkWidthActive:.................4X > LinkSpeedSupported:..............2.5 Gbps > LinkState:.......................Active > PhysLinkState:...................LinkUp > LinkDownDefState:................Polling > ProtectBits:.....................0 > LMC:.............................0 > LinkSpeedActive:.................2.5 Gbps > LinkSpeedEnabled:................2.5 Gbps > NeighborMTU:.....................2048 > SMSL:............................0 > VLCap:...........................VL0-3 > InitType:........................0x00 > VLHighLimit:.....................0 > VLArbHighCap:....................8 > VLArbLowCap:.....................8 > InitReply:.......................0x00 > MtuCap:..........................2048 > VLStallCount:....................7 > HoqLife:.........................0 > OperVLs:.........................VL0-3 > PartEnforceInb:..................0 > PartEnforceOutb:.................0 > FilterRawInb:....................0 > FilterRawOutb:...................0 > MkeyViolations:..................0 > PkeyViolations:..................0 > QkeyViolations:..................0 > GuidCap:.........................32 > ClientReregister:................0 > SubnetTimeout:...................17 > RespTimeVal:.....................16 > LocalPhysErr:....................15 > OverrunErr:......................15 > MaxCreditHint:...................0 > RoundTrip:.......................0 > > > Looks fine. > > > I did some checking, and It's not just this node having problems, all nodes > seem to be having this same problem. > > > Would you also run ibchecknet ? > What error messages are on the SM side ? > -- Hal > > > jeff > Hal Rosenstock wrote: > 2009/3/17 jeffrey Lang : > First let me say, I hope this is the right list for this email, if not > please forgive me. > I have a small 16 node compute cluster.    The university where I work at > recently opened a new Datacenter.  My cluster was moved from the old > Datacenter.   Before the move the inifiniband was working properly, after > the move the ipoib has stopped working. > The cluster runs Centos 4 with all the latest updates and the Centos > distributed OFED code.   My plan was to update the OFED code once things had > restablized. > For the move, I shutdown the cluster, removed the inifiniband cables and the > cluster was moved.   I then reinstalled the infiniband cables (not in the > same order before the move) and brought every thing back up. > When i brought the cluster back up the ipoib would not work.  The only > message in the log file is "Mar 15 04:04:32 h2o01 kernel: ib0: multicast > join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22". > I think that there may be a rate issue in terms of this node relative > to the IPoIB broadcast group which by default is 10 Gbps (4x SDR). > What does this node's portinfo show (smpquery portinfo -D 0) in terms > of link width and speed ? > -- Hal > The master node can see all the systems: > [root at h2o01 log]# ibnodes > Ca    : 0x00066a0098007e99 ports 1 "h2o17 HCA-1" > Ca    : 0x00066a0098007e9b ports 1 "h2o18 HCA-1" > Ca    : 0x00066a0098007e97 ports 1 "h2o16 HCA-1" > Ca    : 0x00066a0098007e8c ports 1 "h2o15 HCA-1" > Ca    : 0x00066a0098007e94 ports 1 "h2o14 HCA-1" > Ca    : 0x00066a0098007e93 ports 1 "h2o13 HCA-1" > Ca    : 0x00066a0098007e8e ports 1 "h2o12 HCA-1" > Ca    : 0x00066a0098007e90 ports 1 "h2o11 HCA-1" > Ca    : 0x00066a0098007e98 ports 1 "h2o10 HCA-1" > Ca    : 0x00066a0098007e95 ports 1 "h2o09 HCA-1" > Ca    : 0x00066a0098007e8f ports 1 "h2o08 HCA-1" > Ca    : 0x00066a0098007e92 ports 1 "h2o07 HCA-1" > Ca    : 0x00066a0098007e8d ports 1 "h2o06 HCA-1" > Ca    : 0x00066a0098007e91 ports 1 "h2o05 HCA-1" > Ca    : 0x00066a0098007e96 ports 1 "h2ocfs HCA-1" > Ca    : 0x00066a0098007e9c ports 1 "h2o01 HCA-1" > Switch    : 0x00066a00d8000593 ports 24 "SilverStorm 9024 > GUID=0x00066a00d8000593" enhanced port 0 lid 1 lmc 0 > I've reset the sm on the switch, but nothing seems to work. > Any ideas of where to look for whats causing the problem? > jeff > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From jrlang at uwyo.edu Wed Mar 18 14:15:28 2009 From: jrlang at uwyo.edu (jeffrey Lang) Date: Wed, 18 Mar 2009 15:15:28 -0600 Subject: [ofa-general] Need help with Infiniband problem In-Reply-To: References: <49BFBA94.5000709@uwyo.edu> <49C01E5B.5070703@uwyo.edu> <49C10730.4050003@uwyo.edu> Message-ID: <49C16470.6040705@uwyo.edu> I'm using the version currently that is released with Centos 4.X which shows as "infiniband-diags-1.3.6-1.el4". found this in the syslog Mar 18 13:47:43 h2o01 saquery[21966]: Unable to Open IBT Device[/dev/SysIbt] Now i just need to figure out why the device entry doesn't exist. With the firmware update the errors below have disappeared and the IPOIB is now working. jeff Hal Rosenstock wrote: > On Wed, Mar 18, 2009 at 10:37 AM, jeffrey Lang wrote: > >> Here's the output for ibchecknet: >> >> [root at h2o01 ~]# ibchecknet >> perfquery: iberror: failed: smp query nodeinfo: Node type not CA >> > > What diags version is being used ? > > >> Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port all: >> FAILED >> #warn: counter SymbolErrors = 43259 (threshold 10) lid 1 port 17 >> Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port 17: >> FAILED >> #warn: counter LinkRecovers = 207 (threshold 10) lid 1 port 2 >> #warn: counter RcvErrors = 112 (threshold 10) lid 1 port 2 >> Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port 2: >> FAILED >> #warn: counter LinkDowned = 10 (threshold 10) lid 1 port 1 >> #warn: counter RcvErrors = 95 (threshold 10) lid 1 port 1 >> Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port 1: >> FAILED >> > > Are the counts for these ports (1,2,17) changing ? You can look with > perfquery 1 . > This is a separate issue from the lack of an IPoIB broadcast group > assuming these numbers are incrementing. > > -- Hal > > >> # Checking Ca: nodeguid 0x00066a0098007e99 >> >> # Checking Ca: nodeguid 0x00066a0098007e9b >> >> # Checking Ca: nodeguid 0x00066a0098007e97 >> >> # Checking Ca: nodeguid 0x00066a0098007e8c >> >> # Checking Ca: nodeguid 0x00066a0098007e94 >> >> # Checking Ca: nodeguid 0x00066a0098007e93 >> >> # Checking Ca: nodeguid 0x00066a0098007e8e >> >> # Checking Ca: nodeguid 0x00066a0098007e90 >> >> # Checking Ca: nodeguid 0x00066a0098007e98 >> >> # Checking Ca: nodeguid 0x00066a0098007e95 >> >> # Checking Ca: nodeguid 0x00066a0098007e8f >> >> # Checking Ca: nodeguid 0x00066a0098007e92 >> >> # Checking Ca: nodeguid 0x00066a0098007e8d >> >> # Checking Ca: nodeguid 0x00066a0098007e91 >> >> # Checking Ca: nodeguid 0x00066a0098007e96 >> >> # Checking Ca: nodeguid 0x00066a0098007e9c >> >> ## Summary: 17 nodes checked, 0 bad nodes found >> ## 32 ports checked, 0 bad ports found >> ## 3 ports have errors beyond threshold >> >> >> >> >> I see these messages in the switch log now: >> >> E|2009/03/18 07:34:28.635S: Thread "esm_sar" (0x83394a90) >> ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of >> 0x0000000000010083 does not have bits required to create a group >> (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for >> request from h2o12 HCA-1, Port 0x00066A00A0007E8E, LID 0x000C, returning >> status 0x0600 : 0 >> >> >> I would have to assume that this is my problem, but how to fix? >> >> jeff >> >> >> >> >> >> >> Hal Rosenstock wrote: >> >> On Tue, Mar 17, 2009 at 6:04 PM, jeffrey Lang wrote: >> >> >> Here's the output smpquery portinfo -D 0 as requested below: >> [root at h2o01 ~]# smpquery portinfo -D 0 >> # Port info: DR path 0 port 0 >> Mkey:............................0x0000000000000000 >> GidPrefix:.......................0xfe80000000000000 >> Lid:.............................0x0003 >> SMLid:...........................0x0001 >> CapMask:.........................0x2510a68 >> IsTrapSupported >> IsAutomaticMigrationSupported >> IsSLMappingSupported >> IsLedInfoSupported >> IsSystemImageGUIDsupported >> IsCommunicatonManagementSupported >> IsVendorClassSupported >> IsCapabilityMaskNoticeSupported >> IsClientRegistrationSupported >> DiagCode:........................0x0000 >> MkeyLeasePeriod:.................0 >> LocalPort:.......................1 >> LinkWidthEnabled:................1X or 4X >> LinkWidthSupported:..............1X or 4X >> LinkWidthActive:.................4X >> LinkSpeedSupported:..............2.5 Gbps >> LinkState:.......................Active >> PhysLinkState:...................LinkUp >> LinkDownDefState:................Polling >> ProtectBits:.....................0 >> LMC:.............................0 >> LinkSpeedActive:.................2.5 Gbps >> LinkSpeedEnabled:................2.5 Gbps >> NeighborMTU:.....................2048 >> SMSL:............................0 >> VLCap:...........................VL0-3 >> InitType:........................0x00 >> VLHighLimit:.....................0 >> VLArbHighCap:....................8 >> VLArbLowCap:.....................8 >> InitReply:.......................0x00 >> MtuCap:..........................2048 >> VLStallCount:....................7 >> HoqLife:.........................0 >> OperVLs:.........................VL0-3 >> PartEnforceInb:..................0 >> PartEnforceOutb:.................0 >> FilterRawInb:....................0 >> FilterRawOutb:...................0 >> MkeyViolations:..................0 >> PkeyViolations:..................0 >> QkeyViolations:..................0 >> GuidCap:.........................32 >> ClientReregister:................0 >> SubnetTimeout:...................17 >> RespTimeVal:.....................16 >> LocalPhysErr:....................15 >> OverrunErr:......................15 >> MaxCreditHint:...................0 >> RoundTrip:.......................0 >> >> >> Looks fine. >> >> >> I did some checking, and It's not just this node having problems, all nodes >> seem to be having this same problem. >> >> >> Would you also run ibchecknet ? >> What error messages are on the SM side ? >> -- Hal >> >> >> jeff >> Hal Rosenstock wrote: >> 2009/3/17 jeffrey Lang : >> First let me say, I hope this is the right list for this email, if not >> please forgive me. >> I have a small 16 node compute cluster. The university where I work at >> recently opened a new Datacenter. My cluster was moved from the old >> Datacenter. Before the move the inifiniband was working properly, after >> the move the ipoib has stopped working. >> The cluster runs Centos 4 with all the latest updates and the Centos >> distributed OFED code. My plan was to update the OFED code once things had >> restablized. >> For the move, I shutdown the cluster, removed the inifiniband cables and the >> cluster was moved. I then reinstalled the infiniband cables (not in the >> same order before the move) and brought every thing back up. >> When i brought the cluster back up the ipoib would not work. The only >> message in the log file is "Mar 15 04:04:32 h2o01 kernel: ib0: multicast >> join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22". >> I think that there may be a rate issue in terms of this node relative >> to the IPoIB broadcast group which by default is 10 Gbps (4x SDR). >> What does this node's portinfo show (smpquery portinfo -D 0) in terms >> of link width and speed ? >> -- Hal >> The master node can see all the systems: >> [root at h2o01 log]# ibnodes >> Ca : 0x00066a0098007e99 ports 1 "h2o17 HCA-1" >> Ca : 0x00066a0098007e9b ports 1 "h2o18 HCA-1" >> Ca : 0x00066a0098007e97 ports 1 "h2o16 HCA-1" >> Ca : 0x00066a0098007e8c ports 1 "h2o15 HCA-1" >> Ca : 0x00066a0098007e94 ports 1 "h2o14 HCA-1" >> Ca : 0x00066a0098007e93 ports 1 "h2o13 HCA-1" >> Ca : 0x00066a0098007e8e ports 1 "h2o12 HCA-1" >> Ca : 0x00066a0098007e90 ports 1 "h2o11 HCA-1" >> Ca : 0x00066a0098007e98 ports 1 "h2o10 HCA-1" >> Ca : 0x00066a0098007e95 ports 1 "h2o09 HCA-1" >> Ca : 0x00066a0098007e8f ports 1 "h2o08 HCA-1" >> Ca : 0x00066a0098007e92 ports 1 "h2o07 HCA-1" >> Ca : 0x00066a0098007e8d ports 1 "h2o06 HCA-1" >> Ca : 0x00066a0098007e91 ports 1 "h2o05 HCA-1" >> Ca : 0x00066a0098007e96 ports 1 "h2ocfs HCA-1" >> Ca : 0x00066a0098007e9c ports 1 "h2o01 HCA-1" >> Switch : 0x00066a00d8000593 ports 24 "SilverStorm 9024 >> GUID=0x00066a00d8000593" enhanced port 0 lid 1 lmc 0 >> I've reset the sm on the switch, but nothing seems to work. >> Any ideas of where to look for whats causing the problem? >> jeff >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: jrlang.vcf Type: text/x-vcard Size: 311 bytes Desc: not available URL: From hal.rosenstock at gmail.com Wed Mar 18 14:34:16 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 18 Mar 2009 17:34:16 -0400 Subject: ***SPAM*** Re: [ofa-general] Need help with Infiniband problem In-Reply-To: <49C16470.6040705@uwyo.edu> References: <49BFBA94.5000709@uwyo.edu> <49C01E5B.5070703@uwyo.edu> <49C10730.4050003@uwyo.edu> <49C16470.6040705@uwyo.edu> Message-ID: On Wed, Mar 18, 2009 at 5:15 PM, jeffrey Lang wrote: > I'm using the version currently that is released with Centos 4.X which shows > as "infiniband-diags-1.3.6-1.el4".   found this in the syslog > > Mar 18 13:47:43 h2o01 saquery[21966]: Unable to Open IBT Device[/dev/SysIbt] I'm unfamiliar with the Centos port and have never seen a message like that so I haven't a clue. -- Hal > Now i just need to figure out why the device entry doesn't exist. > > With the firmware update the errors below have disappeared and the IPOIB is > now working. > > jeff > > Hal Rosenstock wrote: > > On Wed, Mar 18, 2009 at 10:37 AM, jeffrey Lang wrote: > > > Here's the output for ibchecknet: > [root at h2o01 ~]# ibchecknet > perfquery: iberror: failed: smp query nodeinfo: Node type not CA > > > What diags version is being used ? > > > Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port all: > FAILED > #warn: counter SymbolErrors = 43259     (threshold 10) lid 1 port 17 > Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port 17: > FAILED > #warn: counter LinkRecovers = 207       (threshold 10) lid 1 port 2 > #warn: counter RcvErrors = 112  (threshold 10) lid 1 port 2 > Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port 2: > FAILED > #warn: counter LinkDowned = 10  (threshold 10) lid 1 port 1 > #warn: counter RcvErrors = 95   (threshold 10) lid 1 port 1 > Error check on lid 1 (SilverStorm 9024 GUID=0x00066a00d8000593) port 1: > FAILED > > > Are the counts for these ports (1,2,17) changing ? You can look with > perfquery 1 . > This is a separate issue from the lack of an IPoIB broadcast group > assuming these numbers are incrementing. > -- Hal > > > # Checking Ca: nodeguid 0x00066a0098007e99 > # Checking Ca: nodeguid 0x00066a0098007e9b > # Checking Ca: nodeguid 0x00066a0098007e97 > # Checking Ca: nodeguid 0x00066a0098007e8c > # Checking Ca: nodeguid 0x00066a0098007e94 > # Checking Ca: nodeguid 0x00066a0098007e93 > # Checking Ca: nodeguid 0x00066a0098007e8e > # Checking Ca: nodeguid 0x00066a0098007e90 > # Checking Ca: nodeguid 0x00066a0098007e98 > # Checking Ca: nodeguid 0x00066a0098007e95 > # Checking Ca: nodeguid 0x00066a0098007e8f > # Checking Ca: nodeguid 0x00066a0098007e92 > # Checking Ca: nodeguid 0x00066a0098007e8d > # Checking Ca: nodeguid 0x00066a0098007e91 > # Checking Ca: nodeguid 0x00066a0098007e96 > # Checking Ca: nodeguid 0x00066a0098007e9c > ## Summary: 17 nodes checked, 0 bad nodes found > ##          32 ports checked, 0 bad ports found > ##          3 ports have errors beyond threshold > I see these messages in the switch log now: > E|2009/03/18 07:34:28.635S: Thread "esm_sar" (0x83394a90) > ESM: Embedded SM Error: sa_McMemberRecord_Set: Component mask of > 0x0000000000010083 does not have bits required to create a group > (0x00000000000130C6) for new MGID of 0xFF12401BFFFF0000:00000000FFFFFFFF for > request from h2o12 HCA-1, Port 0x00066A00A0007E8E, LID 0x000C, returning > status 0x0600 : 0 > I would have to assume that this is my problem, but how to fix? > jeff > Hal Rosenstock wrote: > On Tue, Mar 17, 2009 at 6:04 PM, jeffrey Lang wrote: > Here's the output smpquery portinfo -D 0 as requested below: > [root at h2o01 ~]# smpquery portinfo -D 0 > # Port info: DR path 0 port 0 > Mkey:............................0x0000000000000000 > GidPrefix:.......................0xfe80000000000000 > Lid:.............................0x0003 > SMLid:...........................0x0001 > CapMask:.........................0x2510a68 >                 IsTrapSupported >                 IsAutomaticMigrationSupported >                 IsSLMappingSupported >                 IsLedInfoSupported >                 IsSystemImageGUIDsupported >                 IsCommunicatonManagementSupported >                 IsVendorClassSupported >                 IsCapabilityMaskNoticeSupported >                 IsClientRegistrationSupported > DiagCode:........................0x0000 > MkeyLeasePeriod:.................0 > LocalPort:.......................1 > LinkWidthEnabled:................1X or 4X > LinkWidthSupported:..............1X or 4X > LinkWidthActive:.................4X > LinkSpeedSupported:..............2.5 Gbps > LinkState:.......................Active > PhysLinkState:...................LinkUp > LinkDownDefState:................Polling > ProtectBits:.....................0 > LMC:.............................0 > LinkSpeedActive:.................2.5 Gbps > LinkSpeedEnabled:................2.5 Gbps > NeighborMTU:.....................2048 > SMSL:............................0 > VLCap:...........................VL0-3 > InitType:........................0x00 > VLHighLimit:.....................0 > VLArbHighCap:....................8 > VLArbLowCap:.....................8 > InitReply:.......................0x00 > MtuCap:..........................2048 > VLStallCount:....................7 > HoqLife:.........................0 > OperVLs:.........................VL0-3 > PartEnforceInb:..................0 > PartEnforceOutb:.................0 > FilterRawInb:....................0 > FilterRawOutb:...................0 > MkeyViolations:..................0 > PkeyViolations:..................0 > QkeyViolations:..................0 > GuidCap:.........................32 > ClientReregister:................0 > SubnetTimeout:...................17 > RespTimeVal:.....................16 > LocalPhysErr:....................15 > OverrunErr:......................15 > MaxCreditHint:...................0 > RoundTrip:.......................0 > Looks fine. > I did some checking, and It's not just this node having problems, all nodes > seem to be having this same problem. > Would you also run ibchecknet ? > What error messages are on the SM side ? > -- Hal > jeff > Hal Rosenstock wrote: > 2009/3/17 jeffrey Lang : > First let me say, I hope this is the right list for this email, if not > please forgive me. > I have a small 16 node compute cluster.    The university where I work at > recently opened a new Datacenter.  My cluster was moved from the old > Datacenter.   Before the move the inifiniband was working properly, after > the move the ipoib has stopped working. > The cluster runs Centos 4 with all the latest updates and the Centos > distributed OFED code.   My plan was to update the OFED code once things had > restablized. > For the move, I shutdown the cluster, removed the inifiniband cables and the > cluster was moved.   I then reinstalled the infiniband cables (not in the > same order before the move) and brought every thing back up. > When i brought the cluster back up the ipoib would not work.  The only > message in the log file is "Mar 15 04:04:32 h2o01 kernel: ib0: multicast > join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -22". > I think that there may be a rate issue in terms of this node relative > to the IPoIB broadcast group which by default is 10 Gbps (4x SDR). > What does this node's portinfo show (smpquery portinfo -D 0) in terms > of link width and speed ? > -- Hal > The master node can see all the systems: > [root at h2o01 log]# ibnodes > Ca    : 0x00066a0098007e99 ports 1 "h2o17 HCA-1" > Ca    : 0x00066a0098007e9b ports 1 "h2o18 HCA-1" > Ca    : 0x00066a0098007e97 ports 1 "h2o16 HCA-1" > Ca    : 0x00066a0098007e8c ports 1 "h2o15 HCA-1" > Ca    : 0x00066a0098007e94 ports 1 "h2o14 HCA-1" > Ca    : 0x00066a0098007e93 ports 1 "h2o13 HCA-1" > Ca    : 0x00066a0098007e8e ports 1 "h2o12 HCA-1" > Ca    : 0x00066a0098007e90 ports 1 "h2o11 HCA-1" > Ca    : 0x00066a0098007e98 ports 1 "h2o10 HCA-1" > Ca    : 0x00066a0098007e95 ports 1 "h2o09 HCA-1" > Ca    : 0x00066a0098007e8f ports 1 "h2o08 HCA-1" > Ca    : 0x00066a0098007e92 ports 1 "h2o07 HCA-1" > Ca    : 0x00066a0098007e8d ports 1 "h2o06 HCA-1" > Ca    : 0x00066a0098007e91 ports 1 "h2o05 HCA-1" > Ca    : 0x00066a0098007e96 ports 1 "h2ocfs HCA-1" > Ca    : 0x00066a0098007e9c ports 1 "h2o01 HCA-1" > Switch    : 0x00066a00d8000593 ports 24 "SilverStorm 9024 > GUID=0x00066a00d8000593" enhanced port 0 lid 1 lmc 0 > I've reset the sm on the switch, but nothing seems to work. > Any ideas of where to look for whats causing the problem? > jeff > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From or.gerlitz at gmail.com Wed Mar 18 15:46:37 2009 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Thu, 19 Mar 2009 00:46:37 +0200 Subject: ***SPAM*** Re: [ofa-general] possible bugs in IB code usage of min() and max() In-Reply-To: References: Message-ID: <15ddcffd0903181546p4e421a9dg2277ad349a30893c@mail.gmail.com> Roland Dreier wrote: > Did you look at the code in question?  What's your opinion? Oh, I see - its min(A,B) where B=min(C,D) so when evaluating B sparse complains on the shadowing which isn't really a problem - that is false alarm. Or. From rdreier at cisco.com Wed Mar 18 19:45:59 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Mar 2009 19:45:59 -0700 Subject: [ofa-general][PATCH 2/2 V2] mlx4_core: Link sensing support In-Reply-To: <49B3DFA4.4050004@mellanox.co.il> (Yevgeny Petrilin's message of "Sun, 08 Mar 2009 17:09:24 +0200") References: <49B3DFA4.4050004@mellanox.co.il> Message-ID: > + if ((port_type[i] == MLX4_PORT_TYPE_ETH) || > + (port_type[i+1] == MLX4_PORT_TYPE_IB)) seems there is a typo here -- should be && instead of ||. I fixed this by hand but I'm wondering how much testing this version of the patch got... > + dev->caps.supported_type[i] == MLX4_PORT_TYPE_AUTO ? 1 : 0; I never understand why people write (boolean expression) ? 1 : 0; since the boolean expression is already 1 or 0 anyway. I fixed this. Anyway, applied, thanks. From rdreier at cisco.com Wed Mar 18 19:56:06 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Mar 2009 19:56:06 -0700 Subject: [ofa-general][PATCH] mlx4_ib: Unregister IB device prior to performing CLOSE PORT command In-Reply-To: <49BFC313.1030901@mellanox.co.il> (Yevgeny Petrilin's message of "Tue, 17 Mar 2009 17:34:43 +0200") References: <49BFC313.1030901@mellanox.co.il> Message-ID: thanks, applied. From rdreier at cisco.com Wed Mar 18 20:00:29 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Mar 2009 20:00:29 -0700 Subject: [ofa-general] [PATCH v3] RDMA/cxgb3: Handle EEH events for active connections. In-Reply-To: <20090309102014.5738.98445.stgit@build.ogc.int> (Steve Wise's message of "Mon, 09 Mar 2009 05:20:14 -0500") References: <20090309102014.5738.98445.stgit@build.ogc.int> Message-ID: wanted to apply this patch but it seems to be relative to some intermediate state of your tree -- for example: > @@ -701,7 +701,7 @@ static int __cxio_tpt_op(struct cxio_rdev *rdev_p, u32 reset_tpt_entry, > u32 stag_idx; > u32 wptr; > > - if (rdev_p->flags) > + if (cxio_fatal_error(rdev_p)) > return -EIO; where upstream doesn't have the rdev_p->flags test. please resend the full patch. From swise at opengridcomputing.com Wed Mar 18 20:10:41 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 18 Mar 2009 22:10:41 -0500 Subject: [ofa-general] [PATCH v3] RDMA/cxgb3: Handle EEH events for active connections. In-Reply-To: References: <20090309102014.5738.98445.stgit@build.ogc.int> Message-ID: <49C1B7B1.6020108@opengridcomputing.com> Roland Dreier wrote: > wanted to apply this patch but it seems to be relative to some > intermediate state of your tree -- for example: > > > @@ -701,7 +701,7 @@ static int __cxio_tpt_op(struct cxio_rdev *rdev_p, u32 reset_tpt_entry, > > u32 stag_idx; > > u32 wptr; > > > > - if (rdev_p->flags) > > + if (cxio_fatal_error(rdev_p)) > > return -EIO; > > where upstream doesn't have the rdev_p->flags test. please resend the full patch. > It is on top of this series which is upstream somewhere: http://lkml.org/lkml/2009/1/26/196 From swise at opengridcomputing.com Wed Mar 18 20:15:59 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Wed, 18 Mar 2009 22:15:59 -0500 Subject: [ofa-general] [PATCH v3] RDMA/cxgb3: Handle EEH events for active connections. In-Reply-To: <49C1B7B1.6020108@opengridcomputing.com> References: <20090309102014.5738.98445.stgit@build.ogc.int> <49C1B7B1.6020108@opengridcomputing.com> Message-ID: <49C1B8EF.4010305@opengridcomputing.com> >> >> where upstream doesn't have the rdev_p->flags test. please resend >> the full patch. >> > It is on top of this series which is upstream somewhere: > > http://lkml.org/lkml/2009/1/26/196 > these dependent patches are in net-next. From rdreier at cisco.com Wed Mar 18 20:19:38 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 18 Mar 2009 20:19:38 -0700 Subject: [ofa-general] [PATCH v3] RDMA/cxgb3: Handle EEH events for active connections. In-Reply-To: <49C1B8EF.4010305@opengridcomputing.com> (Steve Wise's message of "Wed, 18 Mar 2009 22:15:59 -0500") References: <20090309102014.5738.98445.stgit@build.ogc.int> <49C1B7B1.6020108@opengridcomputing.com> <49C1B8EF.4010305@opengridcomputing.com> Message-ID: > these dependent patches are in net-next. OK I'll hang onto this and merge this into my second batch of things after net-next goes into Linus's tree. From gmpc at sanger.ac.uk Thu Mar 19 02:52:33 2009 From: gmpc at sanger.ac.uk (Guy Coates) Date: Thu, 19 Mar 2009 09:52:33 +0000 Subject: [ofa-general] Licensing inconsistencies in OFED 1.4 source code Message-ID: <49C215E1.309@sanger.ac.uk> Hi all, Can someone clarify under what terms the various bits of the OFED source are distributed? Some of the source tarballs in the OFED 1.4 distribution have inconsistent (or missing) licensing information. Specifically: tvflash: Spec file says BSD+GPLv2, but COPYING is GPLv2 only. qperf: Spec file says BSD+GPLv2, but COPYING is GPLV2 only. ibsim: Spec file says BSD+GPLv2, but COPYING is GPLv2 only. rds-tools: No licensing information included. qlvnictools: Spec file says GPLv2 only, ibvexdmtools/COPYING has GPLv2+BSD, qlgc_vnic_daemon/src has no licensing information. Can the code be updated with the correct information? Cheers, Guy -- Dr. Guy Coates, Informatics System Group The Wellcome Trust Sanger Institute, Hinxton, Cambridge, CB10 1HH, UK Tel: +44 (0)1223 834244 x 6925 Fax: +44 (0)1223 496802 -- The Wellcome Trust Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE. From nicolas.morey-chaisemartin at ext.bull.net Thu Mar 19 03:12:15 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Thu, 19 Mar 2009 11:12:15 +0100 Subject: [ofa-general] Couple of questions about OpenSM Message-ID: <49C21A7F.3090203@ext.bull.net> Hi, I'm actually working on tweaking OpenSM to achieve lower routing time on the Fat-Tree algorithm. First question and an easy one: By default with what optimization options is OpenSM compiled with? Without any specific options using git, there a no -O2 or such so all the inline functions are not inline which make a huge bad impact on performances (few millions calls to osm_switch_get_least_hops not inline consumes over 15% of computing time) Next one and a bit harder: In the Fat-Tree we have a 2D array for hop table (destination lid/port num). Why is this table allocated as we need and not all at once? And is it really necessary to check each time if the lid we use is not greater than max_lid_ho ? The only reason I would see for this is if a new node/switch with a bigger lid was added to the fabric while openSM is routing. In such a case, wouldn't a lock protect the variables so new lid can't appear/disappear while it calculates the routes ? If yes, we could allocate all and skip a lot of checks. We have millions of calls to malloc and memset in osm_switch_set_hops plus tests in get_hops/get_least_hops. This may cost a bit more memory, but easily gain 15% on routing computation time. Regards Nicolas From vlad at lists.openfabrics.org Thu Mar 19 03:21:47 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Thu, 19 Mar 2009 03:21:47 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090319-0200 daily build status Message-ID: <20090319102147.CC656E61051@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From itaib at mellanox.com Thu Mar 19 04:19:58 2009 From: itaib at mellanox.com (Itai Baz) Date: Thu, 19 Mar 2009 13:19:58 +0200 Subject: [ofa-general] [PATCH] libibmad: added support for handling of BM (Baseboard management) MADs - FIXED Message-ID: <200903191319.58378.itaib@mellanox.com> Hi Sasha, This patch adds support for handling of BM (Baseboard management) MADs. I Fixed all of your comments, and resbumit this patch with the fixes Thanks Signed-off-by: Itai Baz --- libibmad/Makefile.am | 3 +- libibmad/include/infiniband/mad.h | 49 ++++++++++++++++ libibmad/src/bm.c | 110 +++++++++++++++++++++++++++++++++++++ libibmad/src/libibmad.map | 1 + libibmad/src/register.c | 2 + 5 files changed, 164 insertions(+), 1 deletions(-) create mode 100644 libibmad/src/bm.c diff --git a/libibmad/Makefile.am b/libibmad/Makefile.am index 4f3ba98..618ed3e 100644 --- a/libibmad/Makefile.am +++ b/libibmad/Makefile.am @@ -15,7 +15,8 @@ endif libibmad_la_SOURCES = src/dump.c src/fields.c src/mad.c src/portid.c \ src/resolve.c src/rpc.c src/sa.c src/smp.c src/gs.c \ - src/serv.c src/register.c src/vendor.c src/mad_internal.h + src/serv.c src/register.c src/vendor.c src/mad_internal.h \ + src/bm.c libibmad_la_LDFLAGS = -version-info $(ibmad_api_version) \ -export-dynamic $(libibmad_version_script) diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h index b8290a7..d643d3d 100644 --- a/libibmad/include/infiniband/mad.h +++ b/libibmad/include/infiniband/mad.h @@ -63,6 +63,11 @@ BEGIN_C_DECLS #define IB_SA_MCM_RECSZ 53 #define IB_SA_PR_RECSZ 64 +#define IB_BM_DATA_OFFS 64 +#define IB_BM_DATA_SZ (IB_MAD_SIZE - IB_BM_DATA_OFFS) +#define IB_BM_BKEY_OFFS 24 +#define IB_BM_BKEY_AND_DATA_SZ (IB_MAD_SIZE - IB_BM_BKEY_OFFS) + enum MAD_CLASSES { IB_SMI_CLASS = 0x1, IB_SMI_DIRECT_CLASS = 0x81, @@ -108,6 +113,14 @@ enum MAD_ATTR_ID { INFORM_INFO = 0x3, }; +enum MAD_STATUS { + IB_MAD_STS_OK = (0 << 2), + IB_MAD_STS_BAD_BASE_VER_OR_CLASS = (1 << 2), + IB_MAD_STS_METHOD_NOT_SUPPORTED = (2 << 2), + IB_MAD_STS_METHOD_ATTR_NOT_SUPPORTED = (3 << 2), + IB_MAD_STS_INV_ATTR_VALUE = (7 << 2), +}; + enum SMI_ATTR_ID { IB_ATTR_NODE_DESC = 0x10, IB_ATTR_NODE_INFO = 0x11, @@ -158,6 +171,29 @@ enum GSI_ATTR_ID { IB_GSI_ATTR_LAST }; +enum BM_ATTR_ID { + IB_BM_ATTR_BKEYINFO = 0x10, + IB_BM_ATTR_WRITE_VPD = 0x20, + IB_BM_ATTR_READ_VPD = 0x21, + IB_BM_ATTR_RESET_IBML = 0x22, + IB_BM_ATTR_SET_MODULE_PM_CONTROL = 0x23, + IB_BM_ATTR_GET_MODULE_PM_CONTROL = 0x24, + IB_BM_ATTR_SET_UNIT_PM_CONTROL = 0x25, + IB_BM_ATTR_GET_UNIT_PM_CONTROL = 0x26, + IB_BM_ATTR_SET_IOC_PM_CONTROL = 0x27, + IB_BM_ATTR_GET_IOC_PM_CONTROL = 0x28, + IB_BM_ATTR_SET_MODULE_STATE = 0x29, + IB_BM_ATTR_SET_MODULE_ATTENTION = 0x2A, + IB_BM_ATTR_GET_MODULE_STATUS = 0x2B, + IB_BM_ATTR_IB2IBML = 0x2C, + IB_BM_ATTR_IB2CME = 0x2D, + IB_BM_ATTR_IB2MME = 0x2E, + IB_BM_ATTR_OEM = 0x2F, + + IB_BM_ATTR_LAST +}; + + #define IB_VENDOR_OPENIB_PING_CLASS (IB_VENDOR_RANGE2_START_CLASS + 2) #define IB_VENDOR_OPENIB_SYSSTAT_CLASS (IB_VENDOR_RANGE2_START_CLASS + 3) #define IB_OPENIB_OUI (0x001405) @@ -661,6 +697,16 @@ typedef struct ib_vendor_call { ib_rmpp_hdr_t rmpp; } ib_vendor_call_t; +typedef struct ib_bm_call { + unsigned method; + unsigned attrid; + unsigned mod; + unsigned timeout; + ib_rmpp_hdr_t rmpp; + uint64_t bkey; +} ib_bm_call_t; + + #define IB_MIN_UCAST_LID 1 #define IB_MAX_UCAST_LID (0xc000-1) #define IB_MIN_MCAST_LID 0xc000 @@ -876,6 +922,9 @@ MAD_EXPORT uint8_t *performance_reset_via(void *rcvbuf, ib_portid_t * dest, unsigned timeout, unsigned id, const struct ibmad_port *srcport); +/* bm.c */ +MAD_EXPORT uint8_t * bm_call_via(void *data, ib_portid_t *portid, ib_bm_call_t *call, struct ibmad_port *srcport); + /* dump.c */ MAD_EXPORT ib_mad_dump_fn mad_dump_int, mad_dump_uint, mad_dump_hex, mad_dump_rhex, diff --git a/libibmad/src/bm.c b/libibmad/src/bm.c new file mode 100644 index 0000000..29e0a3e --- /dev/null +++ b/libibmad/src/bm.c @@ -0,0 +1,110 @@ +/* + * Copyright (c) 2004-2007 Voltaire Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ + +#if HAVE_CONFIG_H +# include +#endif /* HAVE_CONFIG_H */ + +#include +#include +#include +#include +#include +#include + +#include + +#undef DEBUG +#define DEBUG if (ibdebug) IBWARN + +static inline int +response_expected(int method) +{ + return method == IB_MAD_METHOD_GET || + method == IB_MAD_METHOD_SET || + method == IB_MAD_METHOD_TRAP; +} + +uint8_t * +bm_call_via(void *data, ib_portid_t *portid, ib_bm_call_t *call, struct ibmad_port *srcport) +{ + ib_rpc_t rpc = {0}; + int resp_expected; + char data_with_bkey[IB_BM_BKEY_AND_DATA_SZ] = {0}; + + DEBUG("route %s data %p", portid2str(portid), data); + if (portid->lid <= 0) { + IBWARN("only lid routes are supported"); + return 0; + } + + resp_expected = response_expected(call->method); + + rpc.mgtclass = IB_BOARD_MGMT_CLASS; + + rpc.method = call->method; + rpc.attr.id = call->attrid; + rpc.attr.mod = call->mod; + rpc.timeout = resp_expected ? call->timeout : 0; + // send data and bkey + rpc.datasz = IB_BM_BKEY_AND_DATA_SZ; + rpc.dataoffs = IB_BM_BKEY_OFFS; + + // copy data to a buffer which also includes the bkey + *((uint64_t *) data_with_bkey) = htonll(call->bkey); + memcpy(data_with_bkey + IB_BM_DATA_OFFS - IB_BM_BKEY_OFFS, data, IB_BM_DATA_SZ); + + DEBUG("method 0x%x attr 0x%x mod 0x%x datasz %d off %d res_ex %d bkey 0x%08x%08x", + rpc.method, rpc.attr.id, rpc.attr.mod, + rpc.datasz, rpc.dataoffs, resp_expected, + (int) (call->bkey >> 32), (int) call->bkey); + + portid->qp = 1; + if (!portid->qkey) + portid->qkey = IB_DEFAULT_QP1_QKEY; + + if (resp_expected) { + mad_rpc_rmpp(srcport, &rpc, portid, 0, data_with_bkey); /* FIXME: no RMPP for now */ + memcpy(data, data_with_bkey + IB_BM_DATA_OFFS - IB_BM_BKEY_OFFS, IB_BM_DATA_SZ); + return data; + } + + if (mad_send_via(&rpc, portid, 0, data_with_bkey, srcport) < 0) { + return 0; + } + else { + memcpy(data, data_with_bkey + IB_BM_DATA_OFFS - IB_BM_BKEY_OFFS, IB_BM_DATA_SZ); + return data; + } +} + diff --git a/libibmad/src/libibmad.map b/libibmad/src/libibmad.map index 4306dbc..0f5f2a0 100644 --- a/libibmad/src/libibmad.map +++ b/libibmad/src/libibmad.map @@ -102,5 +102,6 @@ IBMAD_1.3 { ib_resolve_guid_via; ib_resolve_portid_str_via; ib_resolve_self_via; + bm_call_via; local: *; }; diff --git a/libibmad/src/register.c b/libibmad/src/register.c index eac8f5d..76567e1 100644 --- a/libibmad/src/register.c +++ b/libibmad/src/register.c @@ -68,6 +68,8 @@ static int mgmt_class_vers(int mgmt_class) return 1; case IB_CC_CLASS: return 2; + case IB_BOARD_MGMT_CLASS: + return 1; } return 0; -- 1.5.3.7 From orenk at dev.mellanox.co.il Thu Mar 19 05:12:14 2009 From: orenk at dev.mellanox.co.il (Oren Kladnitsky) Date: Thu, 19 Mar 2009 14:12:14 +0200 Subject: [ofa-general] [ANNOUNCE] Yevgeny K is taking the maintenance of the ibutils package Message-ID: <49C2369E.8070307@dev.mellanox.co.il> Hi. Yevgeny Kliteynik is taking the maintenance of the ibutils package from me. Ibutils git: git://git.openfabrics.org/~kliteyn/ibutils.git Thanks, Oren From tziporet at dev.mellanox.co.il Thu Mar 19 05:26:15 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Thu, 19 Mar 2009 14:26:15 +0200 Subject: [ofa-general] Licensing inconsistencies in OFED 1.4 source code In-Reply-To: <49C215E1.309@sanger.ac.uk> References: <49C215E1.309@sanger.ac.uk> Message-ID: <49C239E7.6010608@mellanox.co.il> Guy Coates wrote: I added the maintainers Tziporet > tvflash: Spec file says BSD+GPLv2, but COPYING is GPLv2 only. > Roland from Cisco is the maintainer > qperf: Spec file says BSD+GPLv2, but COPYING is GPLV2 only. > Johann from Qlogic is the maintainer > ibsim: Spec file says BSD+GPLv2, but COPYING is GPLv2 only. > Sasha from Voltaire is the maintainer > rds-tools: No licensing information included. > Andy from Oracle is the maintainer > qlvnictools: Spec file says GPLv2 only, ibvexdmtools/COPYING has GPLv2+BSD, > qlgc_vnic_daemon/src has no licensing information. > Ramachandra K from Qlogic is the maintainer > > Can the code be updated with the correct information? > > Cheers, > > Guy > > From ogerlitz at voltaire.com Thu Mar 19 05:27:25 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 19 Mar 2009 14:27:25 +0200 (IST) Subject: [ofa-general] [PATCH] ib-diag/vendstat: counter-group-info & config-counter-group vendor mads Message-ID: Implement counter-group-info and config-counter-group IS4 vendor mads. The config-counter-group mad should be used to configure a switch port to make counters such as PortXmitDataSL or PortRcvDataSL active. Signed-off-by: Or Gerlitz Index: management/infiniband-diags/src/vendstat.c =================================================================== --- management.orig/infiniband-diags/src/vendstat.c +++ management/infiniband-diags/src/vendstat.c @@ -52,6 +52,8 @@ /* Vendor specific Attribute IDs */ #define IB_MLX_IS3_GENERAL_INFO 0x17 #define IB_MLX_IS3_CONFIG_SPACE_ACCESS 0x50 +#define IB_MLX_IS4_COUNTER_GROUP_INFO 0x90 +#define IB_MLX_IS4_CONFIG_COUNTER_GROUP 0x91 /* Config space addresses */ #define IB_MLX_IS3_PORT_XMIT_WAIT 0x10013C @@ -105,10 +107,15 @@ typedef struct { is3_record_t record[18]; } is3_config_space_t; +void counter_groups_info(ib_portid_t *portid, int port); +void config_counter_groups(ib_portid_t *portid, int port); + static int general_info, xmit_wait = 0; +static int counter_group_info, config_counter_group, cg0, cg1; static int process_opt(void *context, int ch, char *optarg) { + int ret; switch (ch) { case 'N': general_info = 1; @@ -116,6 +123,15 @@ static int process_opt(void *context, in case 'w': xmit_wait = 1; break; + case 'i': + counter_group_info = 1; + break; + case 'c': + config_counter_group = 1; + ret = sscanf(optarg, "%d,%d", &cg0, &cg1); + if (ret != 2) + return -1; + break; default: return -1; } @@ -136,12 +152,17 @@ int main(int argc, char **argv) const struct ibdiag_opt opts[] = { { "N", 'N', 0, NULL, "show IS3 general information"}, { "w", 'w', 0, NULL, "show IS3 port xmit wait counters"}, + { "i", 'i', 0, NULL, "show is4 counter group info"}, + { "c", 'c', 1, "", "set is4 config counter group"}, { 0 } }; - char usage_args[] = ""; + + char usage_args[] = " [port]"; const char *usage_examples[] = { "-N 6\t\t# read IS3 general information", "-w 6\t\t# read IS3 port xmit wait counters", + "-i 6 12\t# read IS4 port 12 counter group info", + "-c 0,1 6 12\t# set IS4 port 12 config counter group for XmitDataSL", NULL }; @@ -167,13 +188,23 @@ int main(int argc, char **argv) IBERROR("can't resolve self port %s", argv[0]); } + if (counter_group_info) { + counter_groups_info(&portid, port); + exit(0); + } + + if (config_counter_group) { + config_counter_groups(&portid, port); + exit(0); + } + + /* These are Mellanox specific vendor MADs */ + /* but vendors change the VendorId so how know for sure ? */ /* Only General Info and Port Xmit Wait Counters */ /* queries are currently supported */ if (!general_info && !xmit_wait) IBERROR("at least one of -N and -w must be specified"); - /* These are Mellanox specific vendor MADs */ - /* but vendors change the VendorId so how know for sure ? */ /* Would need a list of these and it might not be complete */ /* so for right now, punt on this */ @@ -249,3 +280,94 @@ int main(int argc, char **argv) exit(0); } + + +#define COUNTER_GROUPS_NUM 2 + +typedef struct { + uint8_t reserved1[8]; + uint8_t reserved[3]; + uint8_t num_of_counter_groups; + uint32_t group_masks[COUNTER_GROUPS_NUM]; +} is4_counter_group_info_t; + +typedef struct { + uint8_t reserved[3]; + uint8_t group_select; +} is4_group_select_t; + +typedef struct { + uint8_t reserved1[8]; + uint8_t reserved[4]; + is4_group_select_t group_selects[COUNTER_GROUPS_NUM]; +} is4_config_counter_groups_t; + + +void counter_groups_info(ib_portid_t *portid, int port) +{ + char buf[1024]; + ib_vendor_call_t call; + is4_counter_group_info_t *cg_info; + int i, num_cg; + + memset(&call, 0, sizeof(call)); + call.mgmt_class = IB_MLX_VENDOR_CLASS; + call.method = IB_MAD_METHOD_GET; + call.timeout = ibd_timeout; + call.attrid = IB_MLX_IS4_COUNTER_GROUP_INFO; + call.mod = port; + + /* Counter Group Info */ + memset(&buf, 0, sizeof(buf)); + if (!ib_vendor_call_via(&buf, portid, &call, srcport)) + IBERROR("counter group info query"); + + cg_info = (is4_counter_group_info_t *)&buf; + num_cg = cg_info->num_of_counter_groups; + printf("counter_group_info:\n"); + printf("%d counter groups\n", num_cg); + for (i = 0; i < num_cg; i++) + printf("group%d mask %#x\n", i, ntohl(cg_info->group_masks[i])); +} + +/* Group0 counter config values */ +#define IS4_G0_PortXmtDataSL_0_7 0 +#define IS4_G0_PortXmtDataSL_8_15 1 +#define IS4_G0_PortRcvDataSL_0_7 2 + +/* Group1 counter config values */ +#define IS4_G1_PortXmtDataSL_8_15 1 +#define IS4_G1_PortRcvDataSL_0_7 2 +#define IS4_G1_PortRcvDataSL_8_15 8 + +void config_counter_groups(ib_portid_t *portid, int port) +{ + char buf[1024]; + ib_vendor_call_t call; + is4_config_counter_groups_t *cg_config; + + memset(&call, 0, sizeof(call)); + call.mgmt_class = IB_MLX_VENDOR_CLASS; + call.attrid = IB_MLX_IS4_CONFIG_COUNTER_GROUP; + call.timeout = ibd_timeout; + call.mod = port; + /* set config counter groups for groups 0 and 1 */ + call.method = IB_MAD_METHOD_SET; + + memset(&buf, 0, sizeof(buf)); + cg_config = (is4_config_counter_groups_t *)&buf; + + printf("counter_groups_config: configuring group0 %d group1 %d\n", cg0, cg1); + cg_config->group_selects[0].group_select = cg0; + cg_config->group_selects[1].group_select = cg1; + + if (!ib_vendor_call_via(&buf, portid, &call, srcport)) + IBERROR("config counter group set"); + + /* get config counter groups */ + memset(&buf, 0, sizeof(buf)); + call.method = IB_MAD_METHOD_GET; + + if (!ib_vendor_call_via(&buf, portid, &call, srcport)) + IBERROR("config counter group query"); +} From hal.rosenstock at gmail.com Thu Mar 19 06:17:11 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 19 Mar 2009 09:17:11 -0400 Subject: [ofa-general] [PATCH] libibmad: added support for handling of BM (Baseboard management) MADs - FIXED In-Reply-To: <200903191319.58378.itaib@mellanox.com> References: <200903191319.58378.itaib@mellanox.com> Message-ID: On Thu, Mar 19, 2009 at 7:19 AM, Itai Baz wrote: > Hi Sasha, > > This patch adds support for handling of BM (Baseboard management) MADs. > I Fixed all of your comments, and resbumit this patch with the fixes > > Thanks > > Signed-off-by: Itai Baz > --- >  libibmad/Makefile.am              |    3 +- >  libibmad/include/infiniband/mad.h |   49 ++++++++++++++++ >  libibmad/src/bm.c                 |  110 +++++++++++++++++++++++++++++++++++++ >  libibmad/src/libibmad.map         |    1 + >  libibmad/src/register.c           |    2 + >  5 files changed, 164 insertions(+), 1 deletions(-) >  create mode 100644 libibmad/src/bm.c > > diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h > index b8290a7..d643d3d 100644 > --- a/libibmad/include/infiniband/mad.h > +++ b/libibmad/include/infiniband/mad.h > @@ -661,6 +697,16 @@ typedef struct ib_vendor_call { >        ib_rmpp_hdr_t rmpp; >  } ib_vendor_call_t; > > +typedef struct ib_bm_call { > +       unsigned method; > +       unsigned attrid; > +       unsigned mod; > +       unsigned timeout; > +       ib_rmpp_hdr_t rmpp; Is RMPP supported by BM class ? > +       uint64_t bkey; > +} ib_bm_call_t; > + > + >  #define IB_MIN_UCAST_LID       1 >  #define IB_MAX_UCAST_LID       (0xc000-1) >  #define IB_MIN_MCAST_LID       0xc000 > diff --git a/libibmad/src/bm.c b/libibmad/src/bm.c > new file mode 100644 > index 0000000..29e0a3e > --- /dev/null > +++ b/libibmad/src/bm.c > @@ -0,0 +1,110 @@ > +static inline int > +response_expected(int method) > +{ > +       return method == IB_MAD_METHOD_GET || > +               method == IB_MAD_METHOD_SET || > +               method == IB_MAD_METHOD_TRAP; > +} Most BM attributes use send method and response is indicated via bit in AM but that's on the responder rather than requester side. > + > +uint8_t * > +bm_call_via(void *data, ib_portid_t *portid, ib_bm_call_t *call, struct ibmad_port *srcport) > +{ > +       ib_rpc_t rpc = {0}; > +       int resp_expected; > +       char data_with_bkey[IB_BM_BKEY_AND_DATA_SZ] = {0}; > + > +       DEBUG("route %s data %p", portid2str(portid), data); > +       if (portid->lid <= 0) { > +               IBWARN("only lid routes are supported"); > +               return 0; > +       } > + > +       resp_expected = response_expected(call->method); > + > +       rpc.mgtclass = IB_BOARD_MGMT_CLASS; > + > +       rpc.method = call->method; > +       rpc.attr.id = call->attrid; > +       rpc.attr.mod = call->mod; > +       rpc.timeout = resp_expected ? call->timeout : 0; > +       // send data and bkey > +       rpc.datasz = IB_BM_BKEY_AND_DATA_SZ; > +       rpc.dataoffs = IB_BM_BKEY_OFFS; > + > +       // copy data to a buffer which also includes the bkey > +       *((uint64_t *) data_with_bkey) = htonll(call->bkey); > +       memcpy(data_with_bkey + IB_BM_DATA_OFFS - IB_BM_BKEY_OFFS, data, IB_BM_DATA_SZ); > + > +       DEBUG("method 0x%x attr 0x%x mod 0x%x datasz %d off %d res_ex %d bkey 0x%08x%08x", > +               rpc.method, rpc.attr.id, rpc.attr.mod, > +               rpc.datasz, rpc.dataoffs, resp_expected, > +               (int) (call->bkey >> 32), (int) call->bkey); > + > +       portid->qp = 1; > +       if (!portid->qkey) > +               portid->qkey = IB_DEFAULT_QP1_QKEY; > + > +       if (resp_expected) { > +               mad_rpc_rmpp(srcport, &rpc, portid, 0, data_with_bkey);         /* FIXME: no RMPP for now */ Should this just be mad_rpc (no RMPP) ? -- Hal From hal.rosenstock at gmail.com Thu Mar 19 06:18:24 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 19 Mar 2009 09:18:24 -0400 Subject: [ofa-general] ***SPAM*** Re: [ewg] [ANNOUNCE] Yevgeny K is taking the maintenance of the ibutils package In-Reply-To: <49C2369E.8070307@dev.mellanox.co.il> References: <49C2369E.8070307@dev.mellanox.co.il> Message-ID: On Thu, Mar 19, 2009 at 8:12 AM, Oren Kladnitsky wrote: > Hi. > > Yevgeny Kliteynik is taking the maintenance of > the ibutils package from me. Will he be picking up the outstanding ibutils patches or do they need to be resubmitted ? -- Hal > Ibutils git: > git://git.openfabrics.org/~kliteyn/ibutils.git > > Thanks, > Oren > _______________________________________________ > ewg mailing list > ewg at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > From hal.rosenstock at gmail.com Thu Mar 19 06:21:31 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 19 Mar 2009 09:21:31 -0400 Subject: ***SPAM*** Re: [ofa-general] [PATCH] ib-diag/vendstat: counter-group-info & config-counter-group vendor mads In-Reply-To: References: Message-ID: On Thu, Mar 19, 2009 at 8:27 AM, Or Gerlitz wrote: > Implement counter-group-info and config-counter-group IS4 vendor mads. > The config-counter-group mad should be used to configure a switch port > to make counters such as PortXmitDataSL or PortRcvDataSL active. Do these changes persist across an IS4 reboot ? If not, is there a way to do this ? Also, would you also submit a man page update for this ? -- Hal > > Signed-off-by: Or Gerlitz > > Index: management/infiniband-diags/src/vendstat.c > =================================================================== From dorfman.eli at gmail.com Thu Mar 19 06:37:56 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Thu, 19 Mar 2009 15:37:56 +0200 Subject: [ofa-general] ***SPAM*** Re: [PATCH v3 1/2] opensm: setup routing engine when in use and delete when fail In-Reply-To: <20090317133548.GL12557@sashak.voltaire.com> References: <49A6B618.1090300@gmail.com> <49A6B6EB.80700@gmail.com> <20090312132137.GB8818@sashak.voltaire.com> <49B92C27.7060904@gmail.com> <20090312160528.GW8818@sashak.voltaire.com> <49BE0CC4.6030600@gmail.com> <20090317133548.GL12557@sashak.voltaire.com> Message-ID: <49C24AB4.9060505@gmail.com> setup routing engine when in use and delete when fail setup routing engine and allocate resources before use. delete resources when routing algorithm fails. this will save allocation for routing algorithms that are not used. Signed-off-by: Eli Dorfman --- opensm/include/opensm/osm_opensm.h | 6 ++++++ opensm/opensm/osm_opensm.c | 10 ++-------- opensm/opensm/osm_ucast_mgr.c | 17 +++++++++++++++++ 3 files changed, 25 insertions(+), 8 deletions(-) diff --git a/opensm/include/opensm/osm_opensm.h b/opensm/include/opensm/osm_opensm.h index c121be4..8d1b276 100644 --- a/opensm/include/opensm/osm_opensm.h +++ b/opensm/include/opensm/osm_opensm.h @@ -109,6 +109,9 @@ typedef enum _osm_routing_engine_type { } osm_routing_engine_type_t; /***********/ +struct osm_routing_engine; +struct osm_opensm; + /****s* OpenSM: OpenSM/osm_routing_engine * NAME * struct osm_routing_engine @@ -122,6 +125,7 @@ typedef enum _osm_routing_engine_type { struct osm_routing_engine { const char *name; void *context; + int (*setup) (struct osm_routing_engine *re, struct osm_opensm *p_osm); int (*build_lid_matrices) (void *context); int (*ucast_build_fwd_tables) (void *context); void (*ucast_dump_tables) (void *context); @@ -523,5 +527,7 @@ extern volatile unsigned int osm_exit_flag; * Set to one to cause all threads to leave *********/ +void osm_update_routing_engines(osm_opensm_t *osm, const char *engine_names); + END_C_DECLS #endif /* _OSM_OPENSM_H_ */ diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c index 50d1349..9739122 100644 --- a/opensm/opensm/osm_opensm.c +++ b/opensm/opensm/osm_opensm.c @@ -169,14 +169,7 @@ static void setup_routing_engine(osm_opensm_t *osm, const char *name) memset(re, 0, sizeof(struct osm_routing_engine)); re->name = m->name; - if (m->setup(re, osm)) { - OSM_LOG(&osm->log, OSM_LOG_VERBOSE, - "setup of routing" - " engine \'%s\' failed\n", name); - return; - } - OSM_LOG(&osm->log, OSM_LOG_DEBUG, - "\'%s\' routing engine set up\n", re->name); + re->setup = m->setup; append_routing_engine(osm, re); return; } @@ -236,6 +229,7 @@ static void destroy_routing_engines(osm_opensm_t *osm) r->delete(r->context); free(r); } + osm->routing_engine_list = NULL; } /********************************************************************** diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c index fe0a446..102ffc6 100644 --- a/opensm/opensm/osm_ucast_mgr.c +++ b/opensm/opensm/osm_ucast_mgr.c @@ -970,8 +970,25 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) p_osm->routing_engine_used = OSM_ROUTING_ENGINE_TYPE_NONE; while (p_routing_eng) { + if (!p_routing_eng->context && + p_routing_eng->setup(p_routing_eng, p_osm)) { + OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, + "ERR 3A0F: setup of routing engine \'%s\' failed\n", + p_routing_eng->name); + p_routing_eng = p_routing_eng->next; + continue; + } + OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, + "\'%s\' routing engine set up\n", p_routing_eng->name); + if (!ucast_mgr_route(p_routing_eng, p_osm)) break; + + /* delete unused routing engine */ + if (p_routing_eng->delete) + p_routing_eng->delete(p_routing_eng->context); + + p_routing_eng->context = NULL; p_routing_eng = p_routing_eng->next; } -- 1.5.5 From kliteyn at dev.mellanox.co.il Thu Mar 19 06:33:59 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Thu, 19 Mar 2009 15:33:59 +0200 Subject: [ofa-general] Re: [PATCH RESEND] ibutils: remove -libcommon linkage flag In-Reply-To: <20090215074542.GA7189@sashak.voltaire.com> References: <20090215074542.GA7189@sashak.voltaire.com> Message-ID: <49C249C7.9070204@dev.mellanox.co.il> Sasha Khapyorsky wrote: > Remove -libcommon linkage flag - libibumad doesn't depend from > libibcommon anymore and libibcommon will be removed from management tree > soon. Applied. Thanks -- Yevgeny > Signed-off-by: Sasha Khapyorsky > --- From dorfman.eli at gmail.com Thu Mar 19 06:41:56 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Thu, 19 Mar 2009 15:41:56 +0200 Subject: [ofa-general] Re: [PATCH v3 2/2] opensm: support routing engine update In-Reply-To: <20090317133548.GL12557@sashak.voltaire.com> References: <49A6B618.1090300@gmail.com> <49A6B6EB.80700@gmail.com> <20090312132137.GB8818@sashak.voltaire.com> <49B92C27.7060904@gmail.com> <20090312160528.GW8818@sashak.voltaire.com> <49BE0CC4.6030600@gmail.com> <20090317133548.GL12557@sashak.voltaire.com> Message-ID: <49C24BA4.30602@gmail.com> support routing engine update. save used routing engine as last routing engine. destroy old routing engine list and allocate new routing list. cleanup used routing engine allocation if needed and only after new routing engine was configured. Signed-off-by: Eli Dorfman --- opensm/include/opensm/osm_opensm.h | 1 + opensm/opensm/osm_opensm.c | 49 ++++++++++++++++++++++++++++++++++++ opensm/opensm/osm_subnet.c | 7 ++++- opensm/opensm/osm_ucast_mgr.c | 13 +++++++++ 4 files changed, 69 insertions(+), 1 deletions(-) diff --git a/opensm/include/opensm/osm_opensm.h b/opensm/include/opensm/osm_opensm.h index 8d1b276..658c717 100644 --- a/opensm/include/opensm/osm_opensm.h +++ b/opensm/include/opensm/osm_opensm.h @@ -187,6 +187,7 @@ typedef struct osm_opensm { cl_dispatcher_t disp; cl_plock_t lock; struct osm_routing_engine *routing_engine_list; + struct osm_routing_engine *last_routing_engine; osm_routing_engine_type_t routing_engine_used; osm_stats_t stats; osm_console_t console; diff --git a/opensm/opensm/osm_opensm.c b/opensm/opensm/osm_opensm.c index 9739122..4e95fcf 100644 --- a/opensm/opensm/osm_opensm.c +++ b/opensm/opensm/osm_opensm.c @@ -232,6 +232,55 @@ static void destroy_routing_engines(osm_opensm_t *osm) osm->routing_engine_list = NULL; } +static void update_routing_engine( + struct osm_routing_engine *cur, + struct osm_routing_engine *last) +{ + struct osm_routing_engine *next = cur->next; + + if (!last) + return; /* no last routing engine */ + + memcpy(cur, last, sizeof(*cur)); + /* restore next */ + cur->next = next; +} + +void osm_update_routing_engines(osm_opensm_t *osm, const char *engine_names) +{ + struct osm_routing_engine *r, *l; + + /* find used routing engine and save as last */ + l = r = osm->routing_engine_list; + if (r && osm->routing_engine_used == osm_routing_engine_type(r->name)) { + osm->last_routing_engine = r; + osm->routing_engine_list = r->next; + } + else while ((r = r->next)) { + if (osm->routing_engine_used == + osm_routing_engine_type(r->name)) { + osm->last_routing_engine = r; + l->next = r->next; + break; + } + l = r; + } + /* cleanup prev routing engine list and replace with current list */ + destroy_routing_engines(osm); + setup_routing_engines(osm, engine_names); + + /* check if last routing engine exist in new list and update callbacks */ + r = osm->routing_engine_list; + while (r) { + if (osm->routing_engine_used == + osm_routing_engine_type(r->name)) { + update_routing_engine(r, osm->last_routing_engine); + break; + } + r = r->next; + } +} + /********************************************************************** **********************************************************************/ static void destroy_plugins(osm_opensm_t *osm) diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c index ec15f8a..e308515 100644 --- a/opensm/opensm/osm_subnet.c +++ b/opensm/opensm/osm_subnet.c @@ -151,6 +151,11 @@ static void opts_setup_sm_priority(osm_subn_t *p_subn, void *p_val) osm_set_sm_priority(p_sm, sm_priority); } +static void opts_setup_routing_engine(osm_subn_t *p_subn, void *p_val) +{ + osm_update_routing_engines(p_subn->p_osm, p_val); +} + static void opts_parse_net64(IN osm_subn_t *p_subn, IN char *p_key, IN char *p_val_str, void *p_v1, void *p_v2, void (*pfn)(osm_subn_t *, void *)) @@ -323,7 +328,7 @@ static const opt_rec_t opt_tbl[] = { { "hop_weights_file", OPT_OFFSET(hop_weights_file), opts_parse_charp, NULL, 0 }, { "port_profile_switch_nodes", OPT_OFFSET(port_profile_switch_nodes), opts_parse_boolean, NULL, 1 }, { "sweep_on_trap", OPT_OFFSET(sweep_on_trap), opts_parse_boolean, NULL, 1 }, - { "routing_engine", OPT_OFFSET(routing_engine_names), opts_parse_charp, NULL, 0 }, + { "routing_engine", OPT_OFFSET(routing_engine_names), opts_parse_charp, opts_setup_routing_engine, 1 }, { "connect_roots", OPT_OFFSET(connect_roots), opts_parse_boolean, NULL, 1 }, { "use_ucast_cache", OPT_OFFSET(use_ucast_cache), opts_parse_boolean, NULL, 1 }, { "log_file", OPT_OFFSET(log_file), opts_parse_charp, NULL, 0 }, diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c index 102ffc6..349c4bb 100644 --- a/opensm/opensm/osm_ucast_mgr.c +++ b/opensm/opensm/osm_ucast_mgr.c @@ -999,6 +999,19 @@ int osm_ucast_mgr_process(IN osm_ucast_mgr_t * const p_mgr) p_osm->routing_engine_used = OSM_ROUTING_ENGINE_TYPE_MINHOP; } + /* if for some reason different routing engine is used */ + /* cleanup last unused routing engine */ + p_routing_eng = p_osm->last_routing_engine; + if (p_routing_eng) { + if (p_routing_eng->context && + p_routing_eng->delete && + p_osm->routing_engine_used != + osm_routing_engine_type(p_routing_eng->name)) + p_routing_eng->delete(p_routing_eng->context); + free(p_routing_eng); + p_osm->last_routing_engine = NULL; + } + OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, "%s tables configured on all switches\n", osm_routing_engine_type_str(p_osm->routing_engine_used)); -- 1.5.5 From kliteyn at dev.mellanox.co.il Thu Mar 19 06:34:42 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Thu, 19 Mar 2009 15:34:42 +0200 Subject: [ofa-general] Re: [PATCH] ibuitls: use osm_config.h file instead of osm_build_id.h In-Reply-To: <20090215074701.GB7189@sashak.voltaire.com> References: <20090215074701.GB7189@sashak.voltaire.com> Message-ID: <49C249F2.7010702@dev.mellanox.co.il> Sasha Khapyorsky wrote: > Use standard osm_config.h file for OpenSM build mode detection rather > than not valid osm_build_id.h junk which will be removed very soon from > OpenSM tree. Applied, thanks. -- Yevgeny > Signed-off-by: Sasha Khapyorsky > --- From kliteyn at dev.mellanox.co.il Thu Mar 19 06:35:17 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Thu, 19 Mar 2009 15:35:17 +0200 Subject: [ofa-general] [PATCH] ibutils/ibis: Prepare for pkeys support in osm_get_all_port_attr In-Reply-To: <20090306021646.GA3934@comcast.net> References: <20090306021646.GA3934@comcast.net> Message-ID: <49C24A15.8010009@dev.mellanox.co.il> Hal Rosenstock wrote: > In order not to break tree, this initialization needs to preceed > the change to the management tree where osm_get_all_port_attr > (at least the umad vendor version) will support pkeys. Applied, thanks. -- Yevgeny > Signed-off-by: Hal Rosenstock > From kliteyn at dev.mellanox.co.il Thu Mar 19 06:38:38 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Thu, 19 Mar 2009 15:38:38 +0200 Subject: [ofa-general] Re: [PATCH v2] ibutils/ibis: link ibis dynamically In-Reply-To: <20090215082540.GC7189@sashak.voltaire.com> References: <20090215082540.GC7189@sashak.voltaire.com> Message-ID: <49C24ADE.1050906@dev.mellanox.co.il> Sasha Khapyorsky wrote: > Otherwise when running against ibsim with libumad2sim.so preloaded it > has two instances (static and dynamic resulted by libumad2sim.so > preloading) of libibumad with different internal initializations, etc., > which makes it impossible to use ibutils in ibsim environment. This patch broke the ibis build. Applied, followed by more fixes. Thanks -- Yevgeny > Signed-off-by: Sasha Khapyorsky > --- From kliteyn at dev.mellanox.co.il Thu Mar 19 06:40:46 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Thu, 19 Mar 2009 15:40:46 +0200 Subject: [ofa-general] Re: [ewg] [ANNOUNCE] Yevgeny K is taking the maintenance of the ibutils package In-Reply-To: References: <49C2369E.8070307@dev.mellanox.co.il> Message-ID: <49C24B5E.6010308@dev.mellanox.co.il> Hi Hal, Hal Rosenstock wrote: > On Thu, Mar 19, 2009 at 8:12 AM, Oren Kladnitsky > wrote: >> Hi. >> >> Yevgeny Kliteynik is taking the maintenance of >> the ibutils package from me. > > Will he be picking up the outstanding ibutils patches or do they need > to be resubmitted ? Are there any other outstanding ibutils patches? -- Yevgeny > -- Hal > >> Ibutils git: >> git://git.openfabrics.org/~kliteyn/ibutils.git >> >> Thanks, >> Oren >> _______________________________________________ >> ewg mailing list >> ewg at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg >> > From ramachandra.kuchimanchi at qlogic.com Thu Mar 19 07:10:52 2009 From: ramachandra.kuchimanchi at qlogic.com (Ramachandra K) Date: Thu, 19 Mar 2009 19:40:52 +0530 Subject: ***SPAM*** Re: [ofa-general] Licensing inconsistencies in OFED 1.4 source code In-Reply-To: <49C239E7.6010608@mellanox.co.il> References: <49C215E1.309@sanger.ac.uk> <49C239E7.6010608@mellanox.co.il> Message-ID: <71d336490903190710q495400fdq1a4954ad4b7a825f@mail.gmail.com> Guy, Tziporet, Thanks for letting us know. We will update the qlvnictools/qlgc_vnic_daemon related licensing information. Regards, Ram On Thu, Mar 19, 2009 at 5:56 PM, Tziporet Koren wrote: > Guy Coates wrote: > I added the maintainers > Tziporet > >> tvflash: Spec file says BSD+GPLv2, but COPYING is GPLv2 only. >> > > Roland from Cisco is the maintainer >> >> qperf:   Spec file says BSD+GPLv2, but COPYING is GPLV2 only. >> > > Johann from Qlogic is the maintainer >> >> ibsim:   Spec file says BSD+GPLv2, but COPYING is GPLv2 only. >> > > Sasha from Voltaire is the maintainer >> >> rds-tools: No licensing information included. >> > > Andy from Oracle is the maintainer >> >> qlvnictools: Spec file says GPLv2 only, ibvexdmtools/COPYING has >> GPLv2+BSD, >> qlgc_vnic_daemon/src has no licensing information. >> > > Ramachandra K from Qlogic is the maintainer > >> >> Can the code be updated with the correct information? >> > >> Cheers, >> >> Guy >> >> > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From itaib at mellanox.com Thu Mar 19 07:25:24 2009 From: itaib at mellanox.com (Itai Baz) Date: Thu, 19 Mar 2009 16:25:24 +0200 Subject: [ofa-general] [PATCH] libibmad: added support for handling of BM (Baseboard management) MADs - FIXED without rmpp Message-ID: <200903191625.24559.itaib@mellanox.com> This patch adds support for handling of BM (Baseboard management) MADs. I checked Hal's comment regarding RMPP, indeed there is no need for it for BM, so I have removed rmpp, and i'm using now mad_rpc Signed-off-by: Itai Baz --- libibmad/Makefile.am | 3 +- libibmad/include/infiniband/mad.h | 48 ++++++++++++++++ libibmad/src/bm.c | 110 +++++++++++++++++++++++++++++++++++++ libibmad/src/libibmad.map | 1 + libibmad/src/register.c | 2 + 5 files changed, 163 insertions(+), 1 deletions(-) create mode 100644 libibmad/src/bm.c diff --git a/libibmad/Makefile.am b/libibmad/Makefile.am index 4f3ba98..618ed3e 100644 --- a/libibmad/Makefile.am +++ b/libibmad/Makefile.am @@ -15,7 +15,8 @@ endif libibmad_la_SOURCES = src/dump.c src/fields.c src/mad.c src/portid.c \ src/resolve.c src/rpc.c src/sa.c src/smp.c src/gs.c \ - src/serv.c src/register.c src/vendor.c src/mad_internal.h + src/serv.c src/register.c src/vendor.c src/mad_internal.h \ + src/bm.c libibmad_la_LDFLAGS = -version-info $(ibmad_api_version) \ -export-dynamic $(libibmad_version_script) diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h index b8290a7..2cd4eca 100644 --- a/libibmad/include/infiniband/mad.h +++ b/libibmad/include/infiniband/mad.h @@ -63,6 +63,11 @@ BEGIN_C_DECLS #define IB_SA_MCM_RECSZ 53 #define IB_SA_PR_RECSZ 64 +#define IB_BM_DATA_OFFS 64 +#define IB_BM_DATA_SZ (IB_MAD_SIZE - IB_BM_DATA_OFFS) +#define IB_BM_BKEY_OFFS 24 +#define IB_BM_BKEY_AND_DATA_SZ (IB_MAD_SIZE - IB_BM_BKEY_OFFS) + enum MAD_CLASSES { IB_SMI_CLASS = 0x1, IB_SMI_DIRECT_CLASS = 0x81, @@ -108,6 +113,14 @@ enum MAD_ATTR_ID { INFORM_INFO = 0x3, }; +enum MAD_STATUS { + IB_MAD_STS_OK = (0 << 2), + IB_MAD_STS_BAD_BASE_VER_OR_CLASS = (1 << 2), + IB_MAD_STS_METHOD_NOT_SUPPORTED = (2 << 2), + IB_MAD_STS_METHOD_ATTR_NOT_SUPPORTED = (3 << 2), + IB_MAD_STS_INV_ATTR_VALUE = (7 << 2), +}; + enum SMI_ATTR_ID { IB_ATTR_NODE_DESC = 0x10, IB_ATTR_NODE_INFO = 0x11, @@ -158,6 +171,29 @@ enum GSI_ATTR_ID { IB_GSI_ATTR_LAST }; +enum BM_ATTR_ID { + IB_BM_ATTR_BKEYINFO = 0x10, + IB_BM_ATTR_WRITE_VPD = 0x20, + IB_BM_ATTR_READ_VPD = 0x21, + IB_BM_ATTR_RESET_IBML = 0x22, + IB_BM_ATTR_SET_MODULE_PM_CONTROL = 0x23, + IB_BM_ATTR_GET_MODULE_PM_CONTROL = 0x24, + IB_BM_ATTR_SET_UNIT_PM_CONTROL = 0x25, + IB_BM_ATTR_GET_UNIT_PM_CONTROL = 0x26, + IB_BM_ATTR_SET_IOC_PM_CONTROL = 0x27, + IB_BM_ATTR_GET_IOC_PM_CONTROL = 0x28, + IB_BM_ATTR_SET_MODULE_STATE = 0x29, + IB_BM_ATTR_SET_MODULE_ATTENTION = 0x2A, + IB_BM_ATTR_GET_MODULE_STATUS = 0x2B, + IB_BM_ATTR_IB2IBML = 0x2C, + IB_BM_ATTR_IB2CME = 0x2D, + IB_BM_ATTR_IB2MME = 0x2E, + IB_BM_ATTR_OEM = 0x2F, + + IB_BM_ATTR_LAST +}; + + #define IB_VENDOR_OPENIB_PING_CLASS (IB_VENDOR_RANGE2_START_CLASS + 2) #define IB_VENDOR_OPENIB_SYSSTAT_CLASS (IB_VENDOR_RANGE2_START_CLASS + 3) #define IB_OPENIB_OUI (0x001405) @@ -661,6 +697,15 @@ typedef struct ib_vendor_call { ib_rmpp_hdr_t rmpp; } ib_vendor_call_t; +typedef struct ib_bm_call { + unsigned method; + unsigned attrid; + unsigned mod; + unsigned timeout; + uint64_t bkey; +} ib_bm_call_t; + + #define IB_MIN_UCAST_LID 1 #define IB_MAX_UCAST_LID (0xc000-1) #define IB_MIN_MCAST_LID 0xc000 @@ -876,6 +921,9 @@ MAD_EXPORT uint8_t *performance_reset_via(void *rcvbuf, ib_portid_t * dest, unsigned timeout, unsigned id, const struct ibmad_port *srcport); +/* bm.c */ +MAD_EXPORT uint8_t * bm_call_via(void *data, ib_portid_t *portid, ib_bm_call_t *call, struct ibmad_port *srcport); + /* dump.c */ MAD_EXPORT ib_mad_dump_fn mad_dump_int, mad_dump_uint, mad_dump_hex, mad_dump_rhex, diff --git a/libibmad/src/bm.c b/libibmad/src/bm.c new file mode 100644 index 0000000..cc551a5 --- /dev/null +++ b/libibmad/src/bm.c @@ -0,0 +1,110 @@ +/* + * Copyright (c) 2004-2007 Voltaire Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ + +#if HAVE_CONFIG_H +# include +#endif /* HAVE_CONFIG_H */ + +#include +#include +#include +#include +#include +#include + +#include + +#undef DEBUG +#define DEBUG if (ibdebug) IBWARN + +static inline int +response_expected(int method) +{ + return method == IB_MAD_METHOD_GET || + method == IB_MAD_METHOD_SET || + method == IB_MAD_METHOD_TRAP; +} + +uint8_t * +bm_call_via(void *data, ib_portid_t *portid, ib_bm_call_t *call, struct ibmad_port *srcport) +{ + ib_rpc_t rpc = {0}; + int resp_expected; + char data_with_bkey[IB_BM_BKEY_AND_DATA_SZ] = {0}; + + DEBUG("route %s data %p", portid2str(portid), data); + if (portid->lid <= 0) { + IBWARN("only lid routes are supported"); + return 0; + } + + resp_expected = response_expected(call->method); + + rpc.mgtclass = IB_BOARD_MGMT_CLASS; + + rpc.method = call->method; + rpc.attr.id = call->attrid; + rpc.attr.mod = call->mod; + rpc.timeout = resp_expected ? call->timeout : 0; + // send data and bkey + rpc.datasz = IB_BM_BKEY_AND_DATA_SZ; + rpc.dataoffs = IB_BM_BKEY_OFFS; + + // copy data to a buffer which also includes the bkey + *((uint64_t *) data_with_bkey) = htonll(call->bkey); + memcpy(data_with_bkey + IB_BM_DATA_OFFS - IB_BM_BKEY_OFFS, data, IB_BM_DATA_SZ); + + DEBUG("method 0x%x attr 0x%x mod 0x%x datasz %d off %d res_ex %d bkey 0x%08x%08x", + rpc.method, rpc.attr.id, rpc.attr.mod, + rpc.datasz, rpc.dataoffs, resp_expected, + (int) (call->bkey >> 32), (int) call->bkey); + + portid->qp = 1; + if (!portid->qkey) + portid->qkey = IB_DEFAULT_QP1_QKEY; + + if (resp_expected) { + mad_rpc(srcport, &rpc, portid, data_with_bkey, data_with_bkey); /* FIXME: no RMPP for now */ + memcpy(data, data_with_bkey + IB_BM_DATA_OFFS - IB_BM_BKEY_OFFS, IB_BM_DATA_SZ); + return data; + } + + if (mad_send_via(&rpc, portid, 0, data_with_bkey, srcport) < 0) { + return 0; + } + else { + memcpy(data, data_with_bkey + IB_BM_DATA_OFFS - IB_BM_BKEY_OFFS, IB_BM_DATA_SZ); + return data; + } +} + diff --git a/libibmad/src/libibmad.map b/libibmad/src/libibmad.map index 4306dbc..0f5f2a0 100644 --- a/libibmad/src/libibmad.map +++ b/libibmad/src/libibmad.map @@ -102,5 +102,6 @@ IBMAD_1.3 { ib_resolve_guid_via; ib_resolve_portid_str_via; ib_resolve_self_via; + bm_call_via; local: *; }; diff --git a/libibmad/src/register.c b/libibmad/src/register.c index eac8f5d..76567e1 100644 --- a/libibmad/src/register.c +++ b/libibmad/src/register.c @@ -68,6 +68,8 @@ static int mgmt_class_vers(int mgmt_class) return 1; case IB_CC_CLASS: return 2; + case IB_BOARD_MGMT_CLASS: + return 1; } return 0; -- 1.5.3.7 From hal.rosenstock at gmail.com Thu Mar 19 07:43:13 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 19 Mar 2009 10:43:13 -0400 Subject: [ofa-general] ***SPAM*** Re: [ewg] [ANNOUNCE] Yevgeny K is taking the maintenance of the ibutils package In-Reply-To: <49C24B5E.6010308@dev.mellanox.co.il> References: <49C2369E.8070307@dev.mellanox.co.il> <49C24B5E.6010308@dev.mellanox.co.il> Message-ID: Hi Yevgeny, On Thu, Mar 19, 2009 at 9:40 AM, Yevgeny Kliteynik wrote: > Hi Hal, > > Hal Rosenstock wrote: >> >> On Thu, Mar 19, 2009 at 8:12 AM, Oren Kladnitsky >> wrote: >>> >>> Hi. >>> >>> Yevgeny Kliteynik is taking the maintenance >>> of >>> the ibutils package from me. >> >> Will he be picking up the outstanding ibutils patches or do they need >> to be resubmitted ? > > Are there any other outstanding ibutils patches? Not AFAIK. Have the changes been pushed to the server ? Thanks. -- Hal > -- Yevgeny > >> -- Hal >> >>> Ibutils git: >>> git://git.openfabrics.org/~kliteyn/ibutils.git >>> >>> Thanks, >>> Oren >>> _______________________________________________ >>> ewg mailing list >>> ewg at lists.openfabrics.org >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg >>> >> > > From hal.rosenstock at gmail.com Thu Mar 19 07:50:24 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 19 Mar 2009 10:50:24 -0400 Subject: [ofa-general] Re: [PATCH] libibmad: added support for handling of BM (Baseboard management) MADs - FIXED without rmpp In-Reply-To: <200903191625.24559.itaib@mellanox.com> References: <200903191625.24559.itaib@mellanox.com> Message-ID: On Thu, Mar 19, 2009 at 10:25 AM, Itai Baz wrote: > This patch adds support for handling of BM (Baseboard management) MADs. > > I checked Hal's comment regarding RMPP, indeed there is no need for it for BM, so I have removed rmpp, and i'm using now mad_rpc Thanks. Also, is the response_expected handling sufficient ? For any BMSend, it will say no response expected. Is that what is really desired ? I'm not sure on what basis the BMA sets the response bit in the AM on a "response" to a BMSend but that is used in the kernel (to determine an incoming BM response). Should this mimic that idea ? -- Hal From kliteyn at dev.mellanox.co.il Thu Mar 19 08:06:35 2009 From: kliteyn at dev.mellanox.co.il (Yevgeny Kliteynik) Date: Thu, 19 Mar 2009 17:06:35 +0200 Subject: [ofa-general] Re: [ewg] [ANNOUNCE] Yevgeny K is taking the maintenance of the ibutils package In-Reply-To: References: <49C2369E.8070307@dev.mellanox.co.il> <49C24B5E.6010308@dev.mellanox.co.il> Message-ID: <49C25F7B.6090102@dev.mellanox.co.il> Hal, Hal Rosenstock wrote: >> Are there any other outstanding ibutils patches? > > Not AFAIK. Have the changes been pushed to the server ? Thanks. Yes, everything is in main trunk. -- Yevgeny From hal.rosenstock at gmail.com Thu Mar 19 08:49:06 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 19 Mar 2009 11:49:06 -0400 Subject: [ofa-general] Re: [PATCH] libibmad: added support for handling of BM (Baseboard management) MADs - FIXED without rmpp In-Reply-To: <5D49E7A8952DC44FB38C38FA0D758EAD021D0EDD@mtlexch01.mtl.com> References: <200903191625.24559.itaib@mellanox.com> <5D49E7A8952DC44FB38C38FA0D758EAD021D0EDD@mtlexch01.mtl.com> Message-ID: On Thu, Mar 19, 2009 at 11:00 AM, Itai Baz wrote: > As far as I'm aware, the response_expected handling is sufficient. > BMGet and BMSet always have response. > BMSend may be used both as a request from client to server, and as a > response from server to client Yes, but is transaction handling (request/response) used or needed on some BMSends ? -- Hal > -----Original Message----- > From: Hal Rosenstock [mailto:hal.rosenstock at gmail.com] > Sent: Thursday, March 19, 2009 4:50 PM > To: Itai Baz > Cc: Sasha Khapyorsky; OpenIB > Subject: Re: [PATCH] libibmad: added support for handling of BM > (Baseboard management) MADs - FIXED without rmpp > > On Thu, Mar 19, 2009 at 10:25 AM, Itai Baz wrote: >> This patch adds support for handling of BM (Baseboard management) > MADs. >> >> I checked Hal's comment regarding RMPP, indeed there is no need for it > >> for BM, so I have removed rmpp, and i'm using now mad_rpc > > Thanks. > > Also, is the response_expected handling sufficient ? For any BMSend, it > will say no response expected. Is that what is really desired ? I'm not > sure on what basis the BMA sets the response bit in the AM on a > "response" to a BMSend but that is used in the kernel (to determine an > incoming BM response). Should this mimic that idea ? > > -- Hal > > > From tziporet at mellanox.co.il Thu Mar 19 09:17:42 2009 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Thu, 19 Mar 2009 18:17:42 +0200 Subject: [ofa-general] OFED 1.4.1-rc2 is available In-Reply-To: <5D49E7A8952DC44FB38C38FA0D758EAD01F50B50@mtlexch01.mtl.com> References: <5D49E7A8952DC44FB38C38FA0D758EAD01F50B50@mtlexch01.mtl.com> Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD021D0FAE@mtlexch01.mtl.com> Hi, OFED-1.4.1-rc2 release is available on http://www.openfabrics.org/downloads/OFED/ofed-1.4.1/OFED-1.4.1-rc2.tgz To get BUILD_ID run ofed_info Please report any issues in bugzilla https://bugs.openfabrics.org/ for OFED 1.4.1 Vladimir & Tziporet ======================================================================== Release information: ------------------------------ Linux Operating Systems: - RedHat EL4 up4: 2.6.9-42.ELsmp * - RedHat EL4 up5: 2.6.9-55.ELsmp - RedHat EL4 up6: 2.6.9-67.ELsmp - RedHat EL4 up7: 2.6.9-78.ELsmp - RedHat EL5: 2.6.18-8.el5 - RedHat EL5 up1: 2.6.18-53.el5 - RedHat EL5 up2: 2.6.18-92.el5 - RedHat EL5 up3: 2.6.18-128.el5 - OEL 4.5: 2.6.9-55.ELsmp - OEL 5.2: 2.6.18-92.el5 - CentOS 5.2: 2.6.18-92.el5 - Fedora C9: 2.6.25-14.fc9 * - SLES10: 2.6.16.21-0.8-smp - SLES10 SP1: 2.6.16.46-0.12-smp - SLES10 SP1 up1: 2.6.16.53-0.16-smp - SLES10 SP2: 2.6.16.60-0.21-smp - SLES11 GA: 2.6.27.13-1-default - OpenSuSE 10.3: 2.6.22.5-31 * - kernel.org: 2.6.26 and 2.6.27 * Minimal QA for these versions Systems: * x86_64 * x86 * ia64 * ppc64 Main Changes from OFED-1.4.1-rc1 ========================== - Fixed issues with compilation on RHEL 5.3 and SLES11 - Fixed RHEL 5.2 and 5.3 backports for NFS/RDMA - Updated bonding package: ib-bonding-0.9.0-38 - Fixed RDS iWARP support - Updated DAPL: compat-dapl-1.2.13 and dapl-2.0.16: v1,v2 - spec files missing Requires(post) statements for sed/coreutils v1,v2 - dtest/dapltest: use $(top_builddir) for .la files during test builds v1,v2 - scm: remove unnecessary thread when using direct objects v2 - scm: remove query gid/lid from connection setup phase v2 - common: add optional 64 bit counters for IA, EP, EVD's (with -DDAPL_COUNTERS) v2 - dtestx: add option -p to print and query new counters v2 - WinOF and OFED: sync up trees for common code base. Updated bonding module: ib-bonding-0.9.0-39 - Low level drivers updated: ipath, nes, mlx4 - Added a module parameter to control number of MTTs per segment in mellanox HCAs (mlx4 & mthca) - mstfling update - 13 Bug fixes (see attached for details) - Attached kernel git tree changes for details Tasks that should be completed for RC3: ============================= 1. High priority bug fixes (we have 16 open now) 2. Open MPI 1.3.1 release 3. MVAPICH 1.1.1.minor release with critical bug fixes 4. OSM release 3.2.6 5. Documentation update -------------- next part -------------- A non-text attachment was scrubbed... Name: ofed-1.4.1-rc1_rc2.log Type: application/octet-stream Size: 31464 bytes Desc: ofed-1.4.1-rc1_rc2.log URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ofed-1.4.1-rc2-fixed-bugs.csv Type: application/octet-stream Size: 1399 bytes Desc: ofed-1.4.1-rc2-fixed-bugs.csv URL: From itaib at mellanox.co.il Thu Mar 19 08:00:23 2009 From: itaib at mellanox.co.il (Itai Baz) Date: Thu, 19 Mar 2009 17:00:23 +0200 Subject: [ofa-general] RE: [PATCH] libibmad: added support for handling of BM (Baseboard management) MADs - FIXED without rmpp In-Reply-To: References: <200903191625.24559.itaib@mellanox.com> Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD021D0EDD@mtlexch01.mtl.com> As far as I'm aware, the response_expected handling is sufficient. BMGet and BMSet always have response. BMSend may be used both as a request from client to server, and as a response from server to client -----Original Message----- From: Hal Rosenstock [mailto:hal.rosenstock at gmail.com] Sent: Thursday, March 19, 2009 4:50 PM To: Itai Baz Cc: Sasha Khapyorsky; OpenIB Subject: Re: [PATCH] libibmad: added support for handling of BM (Baseboard management) MADs - FIXED without rmpp On Thu, Mar 19, 2009 at 10:25 AM, Itai Baz wrote: > This patch adds support for handling of BM (Baseboard management) MADs. > > I checked Hal's comment regarding RMPP, indeed there is no need for it > for BM, so I have removed rmpp, and i'm using now mad_rpc Thanks. Also, is the response_expected handling sufficient ? For any BMSend, it will say no response expected. Is that what is really desired ? I'm not sure on what basis the BMA sets the response bit in the AM on a "response" to a BMSend but that is used in the kernel (to determine an incoming BM response). Should this mimic that idea ? -- Hal From keshetti.mahesh at gmail.com Fri Mar 20 02:15:14 2009 From: keshetti.mahesh at gmail.com (Keshetti Mahesh) Date: Fri, 20 Mar 2009 14:45:14 +0530 Subject: [ofa-general] ***SPAM*** IPoIB problem Message-ID: <829ded920903200215vfe2618fwe584703d7b62c3d3@mail.gmail.com> Hello list, I have isolated an enclosure consisting of 4 nodes each with a Mellanox HCA from an IB cluster containing multiple such enclosures. And connected nodes of the isolated enclosure with another IB switch forming a different subnet. This enclosure uses a same "ethernet" network which is used for the main cluster. Everything (IB related) works fine in the isolated cluster if I don't use IPoIB. When I load IPoIB and run any application using TCP/IP (e.g. ping) on IPoIB interface whole system hangs for 10-15 minutes and strangely all IB related applications (e.g OpenSM) gets killed. ping to the local IPoIB interface works. But ping to any remote IPoIB interface results into above situation. Can anyone please tell me is there anything obviously wrong with the above described configuration? Any help on resolving this issue is greatly appreciated. -Mahesh From vlad at lists.openfabrics.org Fri Mar 20 03:25:10 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Fri, 20 Mar 2009 03:25:10 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090320-0200 daily build status Message-ID: <20090320102510.48CBCE6111E@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From hnrose at comcast.net Fri Mar 20 04:51:46 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Fri, 20 Mar 2009 06:51:46 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] libibmad/sa.c: No need to specify NumbPath field in Get request of SA PathRecord Message-ID: <20090320115146.GA16288@comcast.net> Also, cosmetic formatting changes Signed-off-by: Hal Rosenstock --- diff --git a/libibmad/src/sa.c b/libibmad/src/sa.c index 5174bb5..60c803f 100644 --- a/libibmad/src/sa.c +++ b/libibmad/src/sa.c @@ -45,8 +45,8 @@ #undef DEBUG #define DEBUG if (ibdebug) IBWARN -uint8_t *sa_rpc_call(const struct ibmad_port *ibmad_port, void *rcvbuf, ib_portid_t * portid, - ib_sa_call_t * sa, unsigned timeout) +uint8_t *sa_rpc_call(const struct ibmad_port *ibmad_port, void *rcvbuf, + ib_portid_t * portid, ib_sa_call_t * sa, unsigned timeout) { ib_rpc_t rpc = { 0 }; uint8_t *p; @@ -73,7 +73,7 @@ uint8_t *sa_rpc_call(const struct ibmad_port *ibmad_port, void *rcvbuf, ib_porti if (!portid->qkey) portid->qkey = IB_DEFAULT_QP1_QKEY; - p = mad_rpc_rmpp(ibmad_port, &rpc, portid, 0 /*&sa->rmpp */ , rcvbuf); /* TODO: RMPP */ + p = mad_rpc_rmpp(ibmad_port, &rpc, portid, 0 /*&sa->rmpp */, rcvbuf); /* TODO: RMPP */ sa->recsz = rpc.recsz; @@ -110,8 +110,7 @@ uint8_t *sa_call(void *rcvbuf, ib_portid_t * portid, ib_sa_call_t * sa, #define IB_PR_COMPMASK_PREFERENCE (1ull<<22) #define IB_PR_DEF_MASK (IB_PR_COMPMASK_DGID |\ - IB_PR_COMPMASK_SGID |\ - IB_PR_COMPMASK_NUMBPATH) + IB_PR_COMPMASK_SGID) int ib_path_query_via(const struct ibmad_port *srcport, ibmad_gid_t srcgid, ibmad_gid_t destgid, ib_portid_t * sm_id, void *buf) @@ -130,7 +129,6 @@ int ib_path_query_via(const struct ibmad_port *srcport, ibmad_gid_t srcgid, memset(buf, 0, IB_SA_PR_RECSZ); - mad_encode_field(buf, IB_SA_PR_NPATH_F, &npath); mad_encode_field(buf, IB_SA_PR_DGID_F, destgid); mad_encode_field(buf, IB_SA_PR_SGID_F, srcgid); From sashak at voltaire.com Fri Mar 20 04:12:59 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Mar 2009 13:12:59 +0200 Subject: [ofa-general] Licensing inconsistencies in OFED 1.4 source code In-Reply-To: <49C215E1.309@sanger.ac.uk> References: <49C215E1.309@sanger.ac.uk> Message-ID: <20090320111259.GB6267@sashak.voltaire.com> Hi Guy, On 09:52 Thu 19 Mar , Guy Coates wrote: > > ibsim: Spec file says BSD+GPLv2, but COPYING is GPLv2 only. COPYING file is full text of GPLv2 which is included in package distribution. All source files are distributed under dual license. Sasha From sashak at voltaire.com Fri Mar 20 04:59:41 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Mar 2009 13:59:41 +0200 Subject: [ofa-general] Couple of questions about OpenSM In-Reply-To: <49C21A7F.3090203@ext.bull.net> References: <49C21A7F.3090203@ext.bull.net> Message-ID: <20090320115941.GC6267@sashak.voltaire.com> Hi Nicolas, On 11:12 Thu 19 Mar , Nicolas Morey Chaisemartin wrote: > > First question and an easy one: > By default with what optimization options is OpenSM compiled with? It is defined in autoconf I guess. > Without any specific options using git, there a no -O2 or such so all the inline functions are not inline which make a huge bad impact on performances (few millions calls to osm_switch_get_least_hops not inline consumes over 15% of computing time) In my setup I have in /usr/share/autoconf/autoconf/c.m4: if test "$GCC" = yes; then CFLAGS="-g -O2" And -O2 is turned on by default. > Next one and a bit harder: > In the Fat-Tree we have a 2D array for hop table (destination lid/port num). Why is this table allocated as we need and not all at once? Yes, I think it can be preallocated. Just not that this should not have port arrays for each lid in a fabric, but only for switches. > And is it really necessary to check each time if the lid we use is not greater than max_lid_ho ? May be not, but I will need to check carefully. > The only reason I would see for this is if a new node/switch with a bigger lid was added to the fabric while openSM is routing. In such a case, wouldn't a lock protect the variables so new lid can't appear/disappear while it calculates the routes ? Routing calculation cannot happen in parallel with discovery (it is all serialized in do_sweep() function), we should be protected at least in this part. > If yes, we could allocate all and skip a lot of checks. We have millions of calls to malloc and memset in osm_switch_set_hops plus tests in get_hops/get_least_hops. malloc() calls are conditional, there could be many checks, but only "needed" amount of malloc() itself. > This may cost a bit more memory, Min hop's port arrays preallocation should not cost any extra memory (if done properly - for switches only) - we are allocating all needed buffers in routing calculation time anyway. > but easily gain 15% on routing computation time. Well, I'm skeptical about 15% :). But it doesn't really matter - even 1% performance gain and/or cleaner code would be nice improvement. Sasha From sashak at voltaire.com Fri Mar 20 05:13:58 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Mar 2009 14:13:58 +0200 Subject: [ofa-general] OpenSM/IB management BOF at Sonoma Message-ID: <20090320121358.GD6267@sashak.voltaire.com> Hi, Many people will be in Sonoma next week. Would we like to make informal OpenSM/IB management BOF? I think it was successful experience last year. Sasha From sashak at voltaire.com Fri Mar 20 05:22:11 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Mar 2009 14:22:11 +0200 Subject: [ofa-general] Re: [PATCH] libibmad/sa.c: No need to specify NumbPath field in Get request of SA PathRecord In-Reply-To: <20090320115146.GA16288@comcast.net> References: <20090320115146.GA16288@comcast.net> Message-ID: <20090320122211.GE6267@sashak.voltaire.com> On 06:51 Fri 20 Mar , Hal Rosenstock wrote: > > Also, cosmetic formatting changes It would be really nice to not mix cosmetic and functional changes in one patch. > @@ -110,8 +110,7 @@ uint8_t *sa_call(void *rcvbuf, ib_portid_t * portid, ib_sa_call_t * sa, > #define IB_PR_COMPMASK_PREFERENCE (1ull<<22) > > #define IB_PR_DEF_MASK (IB_PR_COMPMASK_DGID |\ > - IB_PR_COMPMASK_SGID |\ > - IB_PR_COMPMASK_NUMBPATH) > + IB_PR_COMPMASK_SGID) > > int ib_path_query_via(const struct ibmad_port *srcport, ibmad_gid_t srcgid, > ibmad_gid_t destgid, ib_portid_t * sm_id, void *buf) > @@ -130,7 +129,6 @@ int ib_path_query_via(const struct ibmad_port *srcport, ibmad_gid_t srcgid, > > memset(buf, 0, IB_SA_PR_RECSZ); > > - mad_encode_field(buf, IB_SA_PR_NPATH_F, &npath); Then you may want to remove unused npath variable. Also do you think (or know) that it could break some non-OpenSM SAs? Sasha From sashak at voltaire.com Fri Mar 20 05:38:30 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Mar 2009 14:38:30 +0200 Subject: [ofa-general] Re: [PATCH] infiniband-diags/perfquery.c: Label PortXmit/RcvDataSL counters in headings In-Reply-To: <20090313203718.GB6173@comcast.net> References: <20090313203718.GB6173@comcast.net> Message-ID: <20090320123830.GB6689@sashak.voltaire.com> On 15:37 Fri 13 Mar , Hal Rosenstock wrote: > > Also, some cosmetic formatting changes > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From hal.rosenstock at gmail.com Fri Mar 20 05:43:36 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 20 Mar 2009 08:43:36 -0400 Subject: [ofa-general] Re: [PATCH] libibmad/sa.c: No need to specify NumbPath field in Get request of SA PathRecord In-Reply-To: <20090320122211.GE6267@sashak.voltaire.com> References: <20090320115146.GA16288@comcast.net> <20090320122211.GE6267@sashak.voltaire.com> Message-ID: On Fri, Mar 20, 2009 at 8:22 AM, Sasha Khapyorsky wrote: > On 06:51 Fri 20 Mar     , Hal Rosenstock wrote: >> >> Also, cosmetic formatting changes > > It would be really nice to not mix cosmetic and functional changes in > one patch. >> @@ -110,8 +110,7 @@ uint8_t *sa_call(void *rcvbuf, ib_portid_t * portid, ib_sa_call_t * sa, >>  #define      IB_PR_COMPMASK_PREFERENCE                       (1ull<<22) >> >>  #define IB_PR_DEF_MASK (IB_PR_COMPMASK_DGID |\ >> -                     IB_PR_COMPMASK_SGID |\ >> -                     IB_PR_COMPMASK_NUMBPATH) >> +                     IB_PR_COMPMASK_SGID) >> >>  int ib_path_query_via(const struct ibmad_port *srcport, ibmad_gid_t srcgid, >>                     ibmad_gid_t destgid, ib_portid_t * sm_id, void *buf) >> @@ -130,7 +129,6 @@ int ib_path_query_via(const struct ibmad_port *srcport, ibmad_gid_t srcgid, >> >>       memset(buf, 0, IB_SA_PR_RECSZ); >> >> -     mad_encode_field(buf, IB_SA_PR_NPATH_F, &npath); > > Then you may want to remove unused npath variable. Sure; next version shortly. > Also do you think (or know) that it could break some non-OpenSM SAs? It shouldn't (as it's supposed to be ignored by the SA anyhow). -- Hal > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hnrose at comcast.net Fri Mar 20 06:37:21 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Fri, 20 Mar 2009 08:37:21 -0500 Subject: [ofa-general] ***SPAM*** [PATCHv2] libibmad/sa.c: No need to specify NumbPath field in Get request of SA PathRecord Message-ID: <20090320133721.GA21406@comcast.net> Signed-off-by: Hal Rosenstock --- Changes since v1: Removed no longer needed npath variable Removed cosmetic changes diff --git a/libibmad/src/sa.c b/libibmad/src/sa.c index 5174bb5..1a5740b 100644 --- a/libibmad/src/sa.c +++ b/libibmad/src/sa.c @@ -110,18 +110,15 @@ uint8_t *sa_call(void *rcvbuf, ib_portid_t * portid, ib_sa_call_t * sa, #define IB_PR_COMPMASK_PREFERENCE (1ull<<22) #define IB_PR_DEF_MASK (IB_PR_COMPMASK_DGID |\ - IB_PR_COMPMASK_SGID |\ - IB_PR_COMPMASK_NUMBPATH) + IB_PR_COMPMASK_SGID) int ib_path_query_via(const struct ibmad_port *srcport, ibmad_gid_t srcgid, ibmad_gid_t destgid, ib_portid_t * sm_id, void *buf) { - int npath; ib_sa_call_t sa = { 0 }; uint8_t *p; int dlid; - npath = 1; /* only MAD_METHOD_GET is supported */ memset(&sa, 0, sizeof sa); sa.method = IB_MAD_METHOD_GET; sa.attrid = IB_SA_ATTR_PATHRECORD; @@ -130,7 +127,6 @@ int ib_path_query_via(const struct ibmad_port *srcport, ibmad_gid_t srcgid, memset(buf, 0, IB_SA_PR_RECSZ); - mad_encode_field(buf, IB_SA_PR_NPATH_F, &npath); mad_encode_field(buf, IB_SA_PR_DGID_F, destgid); mad_encode_field(buf, IB_SA_PR_SGID_F, srcgid); From nicolas.morey-chaisemartin at ext.bull.net Fri Mar 20 05:59:01 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Fri, 20 Mar 2009 13:59:01 +0100 Subject: [ofa-general] [PATCH] ibim/ibsim.c Fixed ibsim console so it doesn't eat up all CPU when creading from a pipe Message-ID: <49C39315.5070802@ext.bull.net> Signed-off-by: Nicolas Morey-Chaisemartin --- ibsim/ibsim.c | 9 ++++++--- 1 files changed, 6 insertions(+), 3 deletions(-) diff --git a/ibsim/ibsim.c b/ibsim/ibsim.c index e720f0d..5fc3b99 100644 --- a/ibsim/ibsim.c +++ b/ibsim/ibsim.c @@ -597,13 +597,14 @@ static int sim_run_console(int fd) fprintf(simout, "sim%s> ", netstarted ? "" : " (inactive)"); fflush(simout); - return 0; + return ret; } static int sim_run(int con_fd) { fd_set rfds; int i; + int console = 1; socket_basename=getenv("IBSIM_SOCKNAME"); if(!socket_basename) @@ -618,7 +619,8 @@ static int sim_run(int con_fd) for (;;) { FD_ZERO(&rfds); FD_SET(simctl, &rfds); - FD_SET(con_fd, &rfds); + if(console) + FD_SET(con_fd, &rfds); for (i = 0; i < IBSIM_MAX_CLIENTS; i++) if (clients[i].pid) FD_SET(clients[i].fd, &rfds); @@ -634,7 +636,8 @@ static int sim_run(int con_fd) sim_read_pkt(clients[i].fd, i); if (FD_ISSET(con_fd, &rfds)) - sim_run_console(con_fd); + if(sim_run_console(con_fd)<=0) + console = 0; } return 0; -- 1.6.2-rc2.GIT From sashak at voltaire.com Fri Mar 20 06:24:14 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Mar 2009 15:24:14 +0200 Subject: [ofa-general] Re: [PATCH] ibim/ibsim.c Fixed ibsim console so it doesn't eat up all CPU when creading from a pipe In-Reply-To: <49C39315.5070802@ext.bull.net> References: <49C39315.5070802@ext.bull.net> Message-ID: <20090320132414.GC6689@sashak.voltaire.com> On 13:59 Fri 20 Mar , Nicolas Morey Chaisemartin wrote: > @@ -618,7 +619,8 @@ static int sim_run(int con_fd) > for (;;) { > FD_ZERO(&rfds); > FD_SET(simctl, &rfds); > - FD_SET(con_fd, &rfds); > + if(console) > + FD_SET(con_fd, &rfds); > for (i = 0; i < IBSIM_MAX_CLIENTS; i++) > if (clients[i].pid) > FD_SET(clients[i].fd, &rfds); > @@ -634,7 +636,8 @@ static int sim_run(int con_fd) > sim_read_pkt(clients[i].fd, i); > > if (FD_ISSET(con_fd, &rfds)) > - sim_run_console(con_fd); > + if(sim_run_console(con_fd)<=0) > + console = 0; Then one occasional failure will disable the console forever. Wouldn't it be better to analyze HUP state (similar to as it is done with OpenSM console)? Sasha From sashak at voltaire.com Fri Mar 20 06:25:14 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Mar 2009 15:25:14 +0200 Subject: [ofa-general] Re: [PATCHv2] libibmad/sa.c: No need to specify NumbPath field in Get request of SA PathRecord In-Reply-To: <20090320133721.GA21406@comcast.net> References: <20090320133721.GA21406@comcast.net> Message-ID: <20090320132514.GD6689@sashak.voltaire.com> On 08:37 Fri 20 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Fri Mar 20 06:36:33 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Mar 2009 15:36:33 +0200 Subject: [ofa-general] Re: [PATCH] libibmad: Add PortSelect and CounterSelect fields for PortXmit/RcvDataSL attributes In-Reply-To: <20090313203620.GA6173@comcast.net> References: <20090313203620.GA6173@comcast.net> Message-ID: <20090320133633.GE6689@sashak.voltaire.com> On 15:36 Fri 13 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock > --- > diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h > index b8290a7..96bb482 100644 > --- a/libibmad/include/infiniband/mad.h > +++ b/libibmad/include/infiniband/mad.h > @@ -564,7 +564,9 @@ enum MAD_FIELDS { > IB_CPI_TRAP_QKEY_F, > > IB_PC_XMT_DATA_SL_FIRST_F, > - IB_PC_XMT_DATA_SL0_F = IB_PC_XMT_DATA_SL_FIRST_F, > + IB_PC_XMT_DATA_SL_PORT_SELECT_F = IB_PC_XMT_DATA_SL_FIRST_F, > + IB_PC_XMT_DATA_SL_COUNTER_SELECT_F, > + IB_PC_XMT_DATA_SL0_F, > IB_PC_XMT_DATA_SL1_F, > IB_PC_XMT_DATA_SL2_F, > IB_PC_XMT_DATA_SL3_F, > @@ -583,7 +585,9 @@ enum MAD_FIELDS { > IB_PC_XMT_DATA_SL_LAST_F, > > IB_PC_RCV_DATA_SL_FIRST_F, > - IB_PC_RCV_DATA_SL0_F = IB_PC_RCV_DATA_SL_FIRST_F, > + IB_PC_RCV_DATA_SL_PORT_SELECT_F = IB_PC_RCV_DATA_SL_FIRST_F, > + IB_PC_RCV_DATA_SL_COUNTER_SELECT_F, > + IB_PC_RCV_DATA_SL0_F, > IB_PC_RCV_DATA_SL1_F, > IB_PC_RCV_DATA_SL2_F, > IB_PC_RCV_DATA_SL3_F, > diff --git a/libibmad/src/fields.c b/libibmad/src/fields.c > index 19c8fc1..ff94c45 100644 > --- a/libibmad/src/fields.c > +++ b/libibmad/src/fields.c > @@ -402,6 +402,8 @@ static const ib_field_t ib_mad_f[] = { > {BITSOFFS(520, 24), "TrapQP", mad_dump_hex}, > {544, 32, "TrapQKey", mad_dump_hex}, > > + {BITSOFFS(8, 8), "PortSelect", mad_dump_uint}, > + {BITSOFFS(16, 16), "CounterSelect", mad_dump_hex}, > {32, 32, "XmtDataSL0", mad_dump_uint}, > {64, 32, "XmtDataSL1", mad_dump_uint}, > {96, 32, "XmtDataSL2", mad_dump_uint}, > @@ -420,6 +422,8 @@ static const ib_field_t ib_mad_f[] = { > {512, 32, "XmtDataSL15", mad_dump_uint}, > {0, 0}, /* IB_PC_XMT_DATA_SL_LAST_F */ > > + {BITSOFFS(8, 8), "PortSelect", mad_dump_uint}, > + {BITSOFFS(16, 16), "CounterSelect", mad_dump_hex}, What will this actually buy for us? We will just have duplicated entries now. Sasha > {32, 32, "RcvDataSL0", mad_dump_uint}, > {64, 32, "RcvDataSL1", mad_dump_uint}, > {96, 32, "RcvDataSL2", mad_dump_uint}, From hal.rosenstock at gmail.com Fri Mar 20 06:35:56 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 20 Mar 2009 09:35:56 -0400 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] libibmad: Add PortSelect and CounterSelect fields for PortXmit/RcvDataSL attributes In-Reply-To: <20090320133633.GE6689@sashak.voltaire.com> References: <20090313203620.GA6173@comcast.net> <20090320133633.GE6689@sashak.voltaire.com> Message-ID: On Fri, Mar 20, 2009 at 9:36 AM, Sasha Khapyorsky wrote: > On 15:36 Fri 13 Mar     , Hal Rosenstock wrote: >> >> Signed-off-by: Hal Rosenstock >> --- >> diff --git a/libibmad/include/infiniband/mad.h b/libibmad/include/infiniband/mad.h >> index b8290a7..96bb482 100644 >> --- a/libibmad/include/infiniband/mad.h >> +++ b/libibmad/include/infiniband/mad.h >> @@ -564,7 +564,9 @@ enum MAD_FIELDS { >>       IB_CPI_TRAP_QKEY_F, >> >>       IB_PC_XMT_DATA_SL_FIRST_F, >> -     IB_PC_XMT_DATA_SL0_F = IB_PC_XMT_DATA_SL_FIRST_F, >> +     IB_PC_XMT_DATA_SL_PORT_SELECT_F = IB_PC_XMT_DATA_SL_FIRST_F, >> +     IB_PC_XMT_DATA_SL_COUNTER_SELECT_F, >> +     IB_PC_XMT_DATA_SL0_F, >>       IB_PC_XMT_DATA_SL1_F, >>       IB_PC_XMT_DATA_SL2_F, >>       IB_PC_XMT_DATA_SL3_F, >> @@ -583,7 +585,9 @@ enum MAD_FIELDS { >>       IB_PC_XMT_DATA_SL_LAST_F, >> >>       IB_PC_RCV_DATA_SL_FIRST_F, >> -     IB_PC_RCV_DATA_SL0_F = IB_PC_RCV_DATA_SL_FIRST_F, >> +     IB_PC_RCV_DATA_SL_PORT_SELECT_F = IB_PC_RCV_DATA_SL_FIRST_F, >> +     IB_PC_RCV_DATA_SL_COUNTER_SELECT_F, >> +     IB_PC_RCV_DATA_SL0_F, >>       IB_PC_RCV_DATA_SL1_F, >>       IB_PC_RCV_DATA_SL2_F, >>       IB_PC_RCV_DATA_SL3_F, >> diff --git a/libibmad/src/fields.c b/libibmad/src/fields.c >> index 19c8fc1..ff94c45 100644 >> --- a/libibmad/src/fields.c >> +++ b/libibmad/src/fields.c >> @@ -402,6 +402,8 @@ static const ib_field_t ib_mad_f[] = { >>       {BITSOFFS(520, 24), "TrapQP", mad_dump_hex}, >>       {544, 32, "TrapQKey", mad_dump_hex}, >> >> +     {BITSOFFS(8, 8), "PortSelect", mad_dump_uint}, >> +     {BITSOFFS(16, 16), "CounterSelect", mad_dump_hex}, >>       {32,  32, "XmtDataSL0", mad_dump_uint}, >>       {64,  32, "XmtDataSL1", mad_dump_uint}, >>       {96,  32, "XmtDataSL2", mad_dump_uint}, >> @@ -420,6 +422,8 @@ static const ib_field_t ib_mad_f[] = { >>       {512, 32, "XmtDataSL15", mad_dump_uint}, >>       {0, 0},                 /* IB_PC_XMT_DATA_SL_LAST_F */ >> >> +     {BITSOFFS(8, 8), "PortSelect", mad_dump_uint}, >> +     {BITSOFFS(16, 16), "CounterSelect", mad_dump_hex}, > > What will this actually buy for us? Consistency with other PerfMgt attributes for starters. > We will just have duplicated entries now. Not sure what you mean by this. -- Hal > Sasha > >>       {32,  32, "RcvDataSL0", mad_dump_uint}, >>       {64,  32, "RcvDataSL1", mad_dump_uint}, >>       {96,  32, "RcvDataSL2", mad_dump_uint}, > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From sashak at voltaire.com Fri Mar 20 06:43:07 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Mar 2009 15:43:07 +0200 Subject: [ofa-general] Re: OpenSM build now requires lex (flex) and yacc (bison)? In-Reply-To: <20090312133949.2fe26c07.weiny2@llnl.gov> References: <20090312133949.2fe26c07.weiny2@llnl.gov> Message-ID: <20090320134307.GF6689@sashak.voltaire.com> Hi Ira, On 13:39 Thu 12 Mar , Ira Weiny wrote: > diff --git a/opensm/opensm.spec.in b/opensm/opensm.spec.in > index 7b82faf..3f02235 100644 > --- a/opensm/opensm.spec.in > +++ b/opensm/opensm.spec.in > @@ -30,7 +30,7 @@ Group: System Environment/Daemons > URL: http://openfabrics.org/ > Source: http://www.openfabrics.org/downloads/management/@TARBALL@ > BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) > -BuildRequires: libibumad-devel, libtool > +BuildRequires: bison, flex, libibumad-devel, libtool Indeed lex and yacc are needed for OpenSM build now. However different tools can be used (lex, flex, yacc, bison, byacc, etc.). Also some distributions are using other than bison and flex tools. Will this patch limit tool set (and break build on distros which doesn't have bison and flex)? If yes, could this be done using generic requirements? Sasha From sashak at voltaire.com Fri Mar 20 06:46:40 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Mar 2009 15:46:40 +0200 Subject: [ofa-general] Re: [PATCH] libibmad: Add PortSelect and CounterSelect fields for PortXmit/RcvDataSL attributes In-Reply-To: References: <20090313203620.GA6173@comcast.net> <20090320133633.GE6689@sashak.voltaire.com> Message-ID: <20090320134640.GG6689@sashak.voltaire.com> On 09:35 Fri 20 Mar , Hal Rosenstock wrote: > > > We will just have duplicated entries now. > > Not sure what you mean by this. After this patch we will have four identical entries in ib_mad_f[] table: {BITSOFFS(8, 8), "PortSelect", mad_dump_uint}, {BITSOFFS(16, 16), "CounterSelect", mad_dump_hex}, Sasha From hal.rosenstock at gmail.com Fri Mar 20 06:46:18 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 20 Mar 2009 09:46:18 -0400 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] libibmad: Add PortSelect and CounterSelect fields for PortXmit/RcvDataSL attributes In-Reply-To: <20090320134640.GG6689@sashak.voltaire.com> References: <20090313203620.GA6173@comcast.net> <20090320133633.GE6689@sashak.voltaire.com> <20090320134640.GG6689@sashak.voltaire.com> Message-ID: On Fri, Mar 20, 2009 at 9:46 AM, Sasha Khapyorsky wrote: > On 09:35 Fri 20 Mar     , Hal Rosenstock wrote: >> >> > We will just have duplicated entries now. >> >> Not sure what you mean by this. > > After this patch we will have four identical entries in ib_mad_f[] table: > >        {BITSOFFS(8, 8), "PortSelect", mad_dump_uint}, >        {BITSOFFS(16, 16), "CounterSelect", mad_dump_hex}, Do you prefer that they are shared across attributes since the fields are in the same place ? -- Hal > Sasha > From sashak at voltaire.com Fri Mar 20 07:00:29 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Fri, 20 Mar 2009 16:00:29 +0200 Subject: [ofa-general] Re: [PATCH] libibmad: Add PortSelect and CounterSelect fields for PortXmit/RcvDataSL attributes In-Reply-To: References: <20090313203620.GA6173@comcast.net> <20090320133633.GE6689@sashak.voltaire.com> <20090320134640.GG6689@sashak.voltaire.com> Message-ID: <20090320140029.GH6689@sashak.voltaire.com> On 09:46 Fri 20 Mar , Hal Rosenstock wrote: > > Do you prefer that they are shared across attributes since the fields > are in the same place ? This is simpler IMO. Sasha From hal.rosenstock at gmail.com Fri Mar 20 07:07:17 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 20 Mar 2009 10:07:17 -0400 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] libibmad: Add PortSelect and CounterSelect fields for PortXmit/RcvDataSL attributes In-Reply-To: <20090320140029.GH6689@sashak.voltaire.com> References: <20090313203620.GA6173@comcast.net> <20090320133633.GE6689@sashak.voltaire.com> <20090320134640.GG6689@sashak.voltaire.com> <20090320140029.GH6689@sashak.voltaire.com> Message-ID: On Fri, Mar 20, 2009 at 10:00 AM, Sasha Khapyorsky wrote: > On 09:46 Fri 20 Mar     , Hal Rosenstock wrote: >> >> Do you prefer that they are shared across attributes since the fields >> are in the same place ? > > This is simpler IMO. It does eliminate the duplication in ib_mad_f but add additional calls to mad_dump_perfcounters_xmt/rcv_sl. Is that really better ? -- Hal > Sasha > From devel at morey-chaisemartin.com Fri Mar 20 09:03:47 2009 From: devel at morey-chaisemartin.com (Nicolas Morey-Chaisemartin) Date: Fri, 20 Mar 2009 17:03:47 +0100 Subject: [ofa-general] Couple of questions about OpenSM In-Reply-To: <20090320115941.GC6267@sashak.voltaire.com> References: <49C21A7F.3090203@ext.bull.net> <20090320115941.GC6267@sashak.voltaire.com> Message-ID: <49C3BE63.1060703@morey-chaisemartin.com> Sasha Khapyorsky a écrit : > Hi Nicolas, > > On 11:12 Thu 19 Mar , Nicolas Morey Chaisemartin wrote: >> First question and an easy one: >> By default with what optimization options is OpenSM compiled with? > > It is defined in autoconf I guess. > >> Without any specific options using git, there a no -O2 or such so all the inline functions are not inline which make a huge bad impact on performances (few millions calls to osm_switch_get_least_hops not inline consumes over 15% of computing time) > > In my setup I have in /usr/share/autoconf/autoconf/c.m4: > > if test "$GCC" = yes; then > CFLAGS="-g -O2" > > And -O2 is turned on by default. This doesn't seem to be the case for me, I'll check where that comes from > >> Next one and a bit harder: >> In the Fat-Tree we have a 2D array for hop table (destination lid/port num). Why is this table allocated as we need and not all at once? > > Yes, I think it can be preallocated. Just not that this should not have > port arrays for each lid in a fabric, but only for switches. Good. There should be one table for each switch and each of them as one entry per (port/lid) couple no? >> And is it really necessary to check each time if the lid we use is not greater than max_lid_ho ? > > May be not, but I will need to check carefully. > >> The only reason I would see for this is if a new node/switch with a bigger lid was added to the fabric while openSM is routing. In such a case, wouldn't a lock protect the variables so new lid can't appear/disappear while it calculates the routes ? > > Routing calculation cannot happen in parallel with discovery (it is all > serialized in do_sweep() function), we should be protected at least in > this part. > >> If yes, we could allocate all and skip a lot of checks. We have millions of calls to malloc and memset in osm_switch_set_hops plus tests in get_hops/get_least_hops. > > malloc() calls are conditional, there could be many checks, but only > "needed" amount of malloc() itself. > We don't do more than necessary but it costs a lot of time to do millions of syscall. >> This may cost a bit more memory, > > Min hop's port arrays preallocation should not cost any extra memory (if > done properly - for switches only) - we are allocating all needed buffers > in routing calculation time anyway. > >> but easily gain 15% on routing computation time. > > Well, I'm skeptical about 15% :). But it doesn't really matter - even > 1% performance gain and/or cleaner code would be nice improvement. > Well it about 15% of computation time (for Ftree at least) is spent in set_hops and specifically in malloc/memset part If we do it only once at the start, it should probably be much faster ! By the way, is there a reason you don't use likely/unlikely commands in conditions? Regards Nicolas From devel at morey-chaisemartin.com Fri Mar 20 09:04:01 2009 From: devel at morey-chaisemartin.com (Nicolas Morey-Chaisemartin) Date: Fri, 20 Mar 2009 17:04:01 +0100 Subject: [ofa-general] Re: [PATCH] ibim/ibsim.c Fixed ibsim console so it doesn't eat up all CPU when creading from a pipe In-Reply-To: <20090320132414.GC6689@sashak.voltaire.com> References: <49C39315.5070802@ext.bull.net> <20090320132414.GC6689@sashak.voltaire.com> Message-ID: <49C3BE71.205@morey-chaisemartin.com> Sasha Khapyorsky a écrit : > On 13:59 Fri 20 Mar , Nicolas Morey Chaisemartin wrote: >> @@ -618,7 +619,8 @@ static int sim_run(int con_fd) >> for (;;) { >> FD_ZERO(&rfds); >> FD_SET(simctl, &rfds); >> - FD_SET(con_fd, &rfds); >> + if(console) >> + FD_SET(con_fd, &rfds); >> for (i = 0; i < IBSIM_MAX_CLIENTS; i++) >> if (clients[i].pid) >> FD_SET(clients[i].fd, &rfds); >> @@ -634,7 +636,8 @@ static int sim_run(int con_fd) >> sim_read_pkt(clients[i].fd, i); >> >> if (FD_ISSET(con_fd, &rfds)) >> - sim_run_console(con_fd); >> + if(sim_run_console(con_fd)<=0) >> + console = 0; > > Then one occasional failure will disable the console forever. > Wouldn't it be better to analyze HUP state (similar to as it is done > with OpenSM console)? > > Sasha I guess so but in which case would it fail? I'm not really good at poll/select and such so I did it my way ;) We are running hundred thousand of tests using ibsim and it's much faster when not using a whole CPU for itself ! I look for a cleaner way to do this tommorow. Nicolas From devel at morey-chaisemartin.com Fri Mar 20 09:04:51 2009 From: devel at morey-chaisemartin.com (Nicolas Morey-Chaisemartin) Date: Fri, 20 Mar 2009 17:04:51 +0100 Subject: [ofa-general] Re: [PATCH] ibim/ibsim.c Fixed ibsim console so it doesn't eat up all CPU when creading from a pipe In-Reply-To: <20090320132414.GC6689@sashak.voltaire.com> References: <49C39315.5070802@ext.bull.net> <20090320132414.GC6689@sashak.voltaire.com> Message-ID: <49C3BEA3.2050503@morey-chaisemartin.com> Sasha Khapyorsky a écrit : > On 13:59 Fri 20 Mar , Nicolas Morey Chaisemartin wrote: >> @@ -618,7 +619,8 @@ static int sim_run(int con_fd) >> for (;;) { >> FD_ZERO(&rfds); >> FD_SET(simctl, &rfds); >> - FD_SET(con_fd, &rfds); >> + if(console) >> + FD_SET(con_fd, &rfds); >> for (i = 0; i < IBSIM_MAX_CLIENTS; i++) >> if (clients[i].pid) >> FD_SET(clients[i].fd, &rfds); >> @@ -634,7 +636,8 @@ static int sim_run(int con_fd) >> sim_read_pkt(clients[i].fd, i); >> >> if (FD_ISSET(con_fd, &rfds)) >> - sim_run_console(con_fd); >> + if(sim_run_console(con_fd)<=0) >> + console = 0; > > Then one occasional failure will disable the console forever. > Wouldn't it be better to analyze HUP state (similar to as it is done > with OpenSM console)? > > Sasha I guess so but in which case would it fail? I'm not really good at poll/select and such so I did it my way ;) We are running hundred thousand of tests using ibsim and it's much faster when not using a whole CPU for itself ! I look for a cleaner way to do this tommorow. Nicolas From weiny2 at llnl.gov Fri Mar 20 10:05:46 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 20 Mar 2009 10:05:46 -0700 Subject: [ofa-general] OpenSM/IB management BOF at Sonoma In-Reply-To: <20090320121358.GD6267@sashak.voltaire.com> References: <20090320121358.GD6267@sashak.voltaire.com> Message-ID: <20090320100546.2f93de58.weiny2@llnl.gov> I will be there and yes it was successful. Count me in, Ira On Fri, 20 Mar 2009 14:13:58 +0200 Sasha Khapyorsky wrote: > Hi, > > Many people will be in Sonoma next week. Would we like to make informal > OpenSM/IB management BOF? I think it was successful experience last year. > > Sasha > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http:// lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http:// openib.org/mailman/listinfo/openib-general > -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab weiny2 at llnl.gov From weiny2 at llnl.gov Fri Mar 20 10:23:19 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 20 Mar 2009 10:23:19 -0700 Subject: [ofa-general] Re: OpenSM build now requires lex (flex) and yacc (bison)? In-Reply-To: <20090320134307.GF6689@sashak.voltaire.com> References: <20090312133949.2fe26c07.weiny2@llnl.gov> <20090320134307.GF6689@sashak.voltaire.com> Message-ID: <20090320102319.5b2ed182.weiny2@llnl.gov> On Fri, 20 Mar 2009 15:43:07 +0200 Sasha Khapyorsky wrote: > Hi Ira, > > On 13:39 Thu 12 Mar , Ira Weiny wrote: > > diff --git a/opensm/opensm.spec.in b/opensm/opensm.spec.in > > index 7b82faf..3f02235 100644 > > --- a/opensm/opensm.spec.in > > +++ b/opensm/opensm.spec.in > > @@ -30,7 +30,7 @@ Group: System Environment/Daemons > > URL: http:// openfabrics.org/ > > Source: http:// www. openfabrics.org/downloads/management/@TARBALL@ > > BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) > > -BuildRequires: libibumad-devel, libtool > > +BuildRequires: bison, flex, libibumad-devel, libtool > > Indeed lex and yacc are needed for OpenSM build now. However different > tools can be used (lex, flex, yacc, bison, byacc, etc.). Also some > distributions are using other than bison and flex tools. Will this patch > limit tool set (and break build on distros which doesn't have bison and > flex)? If yes, could this be done using generic requirements? I wondered the same thing and was hoping Doug would chime in on how he did it. As far as I know there is no way to generically check. On the other hand I have polled around here and no one knows of a rpm distro which does not provide bison or flex. Perhaps some of the other distros will chime in if this will break them. Ira > > Sasha > -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab weiny2 at llnl.gov From dledford at redhat.com Fri Mar 20 10:31:38 2009 From: dledford at redhat.com (Doug Ledford) Date: Fri, 20 Mar 2009 13:31:38 -0400 Subject: [ofa-general] Re: OpenSM build now requires lex (flex) and yacc (bison)? In-Reply-To: <20090320102319.5b2ed182.weiny2@llnl.gov> References: <20090312133949.2fe26c07.weiny2@llnl.gov> <20090320134307.GF6689@sashak.voltaire.com> <20090320102319.5b2ed182.weiny2@llnl.gov> Message-ID: <1237570298.5106.185.camel@firewall.xsintricity.com> On Fri, 2009-03-20 at 10:23 -0700, Ira Weiny wrote: > On Fri, 20 Mar 2009 15:43:07 +0200 > Sasha Khapyorsky wrote: > > > Hi Ira, > > > > On 13:39 Thu 12 Mar , Ira Weiny wrote: > > > diff --git a/opensm/opensm.spec.in b/opensm/opensm.spec.in > > > index 7b82faf..3f02235 100644 > > > --- a/opensm/opensm.spec.in > > > +++ b/opensm/opensm.spec.in > > > @@ -30,7 +30,7 @@ Group: System Environment/Daemons > > > URL: http:// openfabrics.org/ > > > Source: http:// www. openfabrics.org/downloads/management/@TARBALL@ > > > BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) > > > -BuildRequires: libibumad-devel, libtool > > > +BuildRequires: bison, flex, libibumad-devel, libtool > > > > Indeed lex and yacc are needed for OpenSM build now. However different > > tools can be used (lex, flex, yacc, bison, byacc, etc.). Also some > > distributions are using other than bison and flex tools. Will this patch > > limit tool set (and break build on distros which doesn't have bison and > > flex)? If yes, could this be done using generic requirements? > > I wondered the same thing and was hoping Doug would chime in on how he did it. > As far as I know there is no way to generically check. > > On the other hand I have polled around here and no one knows of a rpm distro > which does not provide bison or flex. Perhaps some of the other distros will > chime in if this will break them. It shouldn't, those are the standards (I think they are even required by the LSB but I could be wrong). I just added the appropriate BuildRequires: entries and went on. > Ira > > > > > Sasha > > > > -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From weiny2 at llnl.gov Fri Mar 20 11:03:28 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 20 Mar 2009 11:03:28 -0700 Subject: [ofa-general] Re: OpenSM build now requires lex (flex) and yacc (bison)? In-Reply-To: <1237570298.5106.185.camel@firewall.xsintricity.com> References: <20090312133949.2fe26c07.weiny2@llnl.gov> <20090320134307.GF6689@sashak.voltaire.com> <20090320102319.5b2ed182.weiny2@llnl.gov> <1237570298.5106.185.camel@firewall.xsintricity.com> Message-ID: <20090320110328.adac7258.weiny2@llnl.gov> On Fri, 20 Mar 2009 13:31:38 -0400 Doug Ledford wrote: > On Fri, 2009-03-20 at 10:23 -0700, Ira Weiny wrote: > > On Fri, 20 Mar 2009 15:43:07 +0200 > > Sasha Khapyorsky wrote: > > > > > Hi Ira, > > > > > > On 13:39 Thu 12 Mar , Ira Weiny wrote: > > > > diff --git a/opensm/opensm.spec.in b/opensm/opensm.spec.in > > > > index 7b82faf..3f02235 100644 > > > > --- a/opensm/opensm.spec.in > > > > +++ b/opensm/opensm.spec.in > > > > @@ -30,7 +30,7 @@ Group: System Environment/Daemons > > > > URL: http:// openfabrics.org/ > > > > Source: http:// www. openfabrics.org/downloads/management/@TARBALL@ > > > > BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) > > > > -BuildRequires: libibumad-devel, libtool > > > > +BuildRequires: bison, flex, libibumad-devel, libtool > > > > > > Indeed lex and yacc are needed for OpenSM build now. However different > > > tools can be used (lex, flex, yacc, bison, byacc, etc.). Also some > > > distributions are using other than bison and flex tools. Will this patch > > > limit tool set (and break build on distros which doesn't have bison and > > > flex)? If yes, could this be done using generic requirements? > > > > I wondered the same thing and was hoping Doug would chime in on how he did it. > > As far as I know there is no way to generically check. > > > > On the other hand I have polled around here and no one knows of a rpm distro > > which does not provide bison or flex. Perhaps some of the other distros will > > chime in if this will break them. > > It shouldn't, those are the standards (I think they are even required by > the LSB but I could be wrong). I just added the appropriate > BuildRequires: entries and went on. > So, just to be clear, applying this patch to the master would make your life easier as well? Ira From dledford at redhat.com Fri Mar 20 11:34:01 2009 From: dledford at redhat.com (Doug Ledford) Date: Fri, 20 Mar 2009 14:34:01 -0400 Subject: [ofa-general] Re: OpenSM build now requires lex (flex) and yacc (bison)? In-Reply-To: <20090320110328.adac7258.weiny2@llnl.gov> References: <20090312133949.2fe26c07.weiny2@llnl.gov> <20090320134307.GF6689@sashak.voltaire.com> <20090320102319.5b2ed182.weiny2@llnl.gov> <1237570298.5106.185.camel@firewall.xsintricity.com> <20090320110328.adac7258.weiny2@llnl.gov> Message-ID: <1237574041.5106.192.camel@firewall.xsintricity.com> On Fri, 2009-03-20 at 11:03 -0700, Ira Weiny wrote: > On Fri, 20 Mar 2009 13:31:38 -0400 > Doug Ledford wrote: > > > On Fri, 2009-03-20 at 10:23 -0700, Ira Weiny wrote: > > > On Fri, 20 Mar 2009 15:43:07 +0200 > > > Sasha Khapyorsky wrote: > > > > > > > Hi Ira, > > > > > > > > On 13:39 Thu 12 Mar , Ira Weiny wrote: > > > > > diff --git a/opensm/opensm.spec.in b/opensm/opensm.spec.in > > > > > index 7b82faf..3f02235 100644 > > > > > --- a/opensm/opensm.spec.in > > > > > +++ b/opensm/opensm.spec.in > > > > > @@ -30,7 +30,7 @@ Group: System Environment/Daemons > > > > > URL: http:// openfabrics.org/ > > > > > Source: http:// www. openfabrics.org/downloads/management/@TARBALL@ > > > > > BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n) > > > > > -BuildRequires: libibumad-devel, libtool > > > > > +BuildRequires: bison, flex, libibumad-devel, libtool > > > > > > > > Indeed lex and yacc are needed for OpenSM build now. However different > > > > tools can be used (lex, flex, yacc, bison, byacc, etc.). Also some > > > > distributions are using other than bison and flex tools. Will this patch > > > > limit tool set (and break build on distros which doesn't have bison and > > > > flex)? If yes, could this be done using generic requirements? > > > > > > I wondered the same thing and was hoping Doug would chime in on how he did it. > > > As far as I know there is no way to generically check. > > > > > > On the other hand I have polled around here and no one knows of a rpm distro > > > which does not provide bison or flex. Perhaps some of the other distros will > > > chime in if this will break them. > > > > It shouldn't, those are the standards (I think they are even required by > > the LSB but I could be wrong). I just added the appropriate > > BuildRequires: entries and went on. > > > > So, just to be clear, applying this patch to the master would make your life > easier as well? Doesn't really matter to me, I use my own spec file. -- Doug Ledford GPG KeyID: CFBFF194 http://people.redhat.com/dledford Infiniband specific RPMs available at http://people.redhat.com/dledford/Infiniband -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 197 bytes Desc: This is a digitally signed message part URL: From hnrose at comcast.net Fri Mar 20 16:12:22 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Fri, 20 Mar 2009 18:12:22 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] infiniband-diags/perfquery.8: Fix typo in short option for PortXmitDataSL counters Message-ID: <20090320231222.GB1424@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/infiniband-diags/man/perfquery.8 b/infiniband-diags/man/perfquery.8 index e0743d6..3a7de48 100644 --- a/infiniband-diags/man/perfquery.8 +++ b/infiniband-diags/man/perfquery.8 @@ -5,7 +5,7 @@ perfquery \- query InfiniBand port counters .SH SYNOPSIS .B perfquery -[\-d(ebug)] [\-G(uid)] [\-x|\-\-extended] [\-X|\-\-xmtsl] [\-R|\-\-rcvsl] [-a(ll_ports)] [-l(oop_ports)] [-r(eset_after_read)] [-R(eset_only)] [\-C ca_name] [\-P ca_port] [\-t(imeout) timeout_ms] [\-V(ersion)] [\-h(elp)] [ [[port] [reset_mask]]] +[\-d(ebug)] [\-G(uid)] [\-x|\-\-extended] [\-X|\-\-xmtsl] [\-S|\-\-rcvsl] [-a(ll_ports)] [-l(oop_ports)] [-r(eset_after_read)] [-R(eset_only)] [\-C ca_name] [\-P ca_port] [\-t(imeout) timeout_ms] [\-V(ersion)] [\-h(elp)] [ [[port] [reset_mask]]] .SH DESCRIPTION .PP From hnrose at comcast.net Fri Mar 20 16:11:01 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Fri, 20 Mar 2009 18:11:01 -0500 Subject: [ofa-general] ***SPAM*** [PATCHv2] libibmad: Add PortSelect and CounterSelect fields for PortXmit/RcvDataSL Message-ID: <20090320231101.GA1424@comcast.net> Signed-off-by: Hal Rosenstock --- Change from v1: Implement via dumping rather than adding additional duplicated fields diff --git a/libibmad/src/dump.c b/libibmad/src/dump.c index e75e178..4f71615 100644 --- a/libibmad/src/dump.c +++ b/libibmad/src/dump.c @@ -701,12 +701,18 @@ void mad_dump_perfcounters_ext(char *buf, int bufsz, void *val, int valsz) void mad_dump_perfcounters_xmt_sl(char *buf, int bufsz, void *val, int valsz) { - _dump_fields(buf, bufsz, val, IB_PC_XMT_DATA_SL_FIRST_F, IB_PC_XMT_DATA_SL_LAST_F); + int cnt; + + cnt = _dump_fields(buf, bufsz, val, IB_PC_EXT_PORT_SELECT_F, IB_PC_EXT_XMT_BYTES_F); + _dump_fields(buf + cnt, bufsz - cnt, val, IB_PC_XMT_DATA_SL_FIRST_F, IB_PC_XMT_DATA_SL_LAST_F); } void mad_dump_perfcounters_rcv_sl(char *buf, int bufsz, void *val, int valsz) { - _dump_fields(buf, bufsz, val, IB_PC_RCV_DATA_SL_FIRST_F, IB_PC_RCV_DATA_SL_LAST_F); + int cnt; + + cnt = _dump_fields(buf, bufsz, val, IB_PC_EXT_PORT_SELECT_F, IB_PC_EXT_XMT_BYTES_F); + _dump_fields(buf + cnt, bufsz - cnt, val, IB_PC_RCV_DATA_SL_FIRST_F, IB_PC_RCV_DATA_SL_LAST_F); } void xdump(FILE * file, char *msg, void *p, int size) From vuhuong at mellanox.com Fri Mar 20 16:13:00 2009 From: vuhuong at mellanox.com (Vu Pham) Date: Fri, 20 Mar 2009 16:13:00 -0700 Subject: [ofa-general] [PATCH] NFS-RDMA fix for connectX fast registration wr, Message-ID: <49C422FC.7040303@mellanox.com> Hi Jon, Here is the patch for the file kernel_patches/fixes/nfsrdma_to_2_6_28_18.patch. This patch will fix the fast registration wr for Mellanox ConnectX. ConnectX FRMR requires local write enable together with remote rdma write enable thanks, -vu -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: nfsrdma_to_2.6.28_13_patch.patch URL: From vuhuong at mellanox.com Fri Mar 20 16:17:56 2009 From: vuhuong at mellanox.com (Vu Pham) Date: Fri, 20 Mar 2009 16:17:56 -0700 Subject: [ofa-general] ofed-1.4.1-rc2 NFS-RDMA server crash, Message-ID: <49C42424.6050103@mellanox.com> Hi Jon, I ran connectathon test -N100 and get this crash on the server. Both server/client are RHEL 5.2 x64 with connectX HCAs Should I open a bug# on bugzilla? thanks, -vu -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: rh5.2_rnfs_server.crash URL: From sashak at voltaire.com Fri Mar 20 17:25:32 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 21 Mar 2009 02:25:32 +0200 Subject: [ofa-general] OpenSM/IB management BOF at Sonoma In-Reply-To: <20090320100546.2f93de58.weiny2@llnl.gov> References: <20090320121358.GD6267@sashak.voltaire.com> <20090320100546.2f93de58.weiny2@llnl.gov> Message-ID: <20090321002532.GA8262@sashak.voltaire.com> On 10:05 Fri 20 Mar , Ira Weiny wrote: > I will be there and yes it was successful. Good! Do you have any date/time preference? Sasha From weiny2 at llnl.gov Fri Mar 20 17:27:18 2009 From: weiny2 at llnl.gov (Ira Weiny) Date: Fri, 20 Mar 2009 17:27:18 -0700 Subject: [ofa-general] OpenSM/IB management BOF at Sonoma In-Reply-To: <20090321002532.GA8262@sashak.voltaire.com> References: <20090320121358.GD6267@sashak.voltaire.com> <20090320100546.2f93de58.weiny2@llnl.gov> <20090321002532.GA8262@sashak.voltaire.com> Message-ID: <20090320172718.af93b06f.weiny2@llnl.gov> If last year is any indication I think it will have to be after the presentations. Perhaps Tues night? I would be willing to skip some presentations perhaps even "OpenFabrics Alliance Annual Meeting" (Tues 5:30) to get started a bit earlier? Anyone else? Ira On Sat, 21 Mar 2009 02:25:32 +0200 Sasha Khapyorsky wrote: > On 10:05 Fri 20 Mar , Ira Weiny wrote: > > I will be there and yes it was successful. > > Good! Do you have any date/time preference? > > Sasha -- Ira Weiny Math Programmer/Computer Scientist Lawrence Livermore National Lab weiny2 at llnl.gov From chu11 at llnl.gov Fri Mar 20 17:47:24 2009 From: chu11 at llnl.gov (Al Chu) Date: Fri, 20 Mar 2009 17:47:24 -0700 Subject: [ofa-general] OpenSM/IB management BOF at Sonoma In-Reply-To: <20090320172718.af93b06f.weiny2@llnl.gov> References: <20090320121358.GD6267@sashak.voltaire.com> <20090320100546.2f93de58.weiny2@llnl.gov> <20090321002532.GA8262@sashak.voltaire.com> <20090320172718.af93b06f.weiny2@llnl.gov> Message-ID: <1237596444.22632.333.camel@auk31.llnl.gov> > Perhaps Tues night? I would be willing to skip some presentations perhaps even "OpenFabrics Alliance Annual Meeting" (Tues 5:30) to get started a bit earlier? Sounds good to me. Al On Fri, 2009-03-20 at 17:27 -0700, Ira Weiny wrote: > If last year is any indication I think it will have to be after the presentations. > > Perhaps Tues night? I would be willing to skip some presentations perhaps even "OpenFabrics Alliance Annual Meeting" (Tues 5:30) to get started a bit earlier? > > Anyone else? > > Ira > > On Sat, 21 Mar 2009 02:25:32 +0200 > Sasha Khapyorsky wrote: > > > On 10:05 Fri 20 Mar , Ira Weiny wrote: > > > I will be there and yes it was successful. > > > > Good! Do you have any date/time preference? > > > > Sasha > > -- Albert Chu chu11 at llnl.gov Computer Scientist High Performance Systems Division Lawrence Livermore National Laboratory From sashak at voltaire.com Fri Mar 20 18:27:32 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 21 Mar 2009 03:27:32 +0200 Subject: [ofa-general] Re: [PATCH] libibmad: Add PortSelect and CounterSelect fields for PortXmit/RcvDataSL attributes In-Reply-To: References: <20090313203620.GA6173@comcast.net> <20090320133633.GE6689@sashak.voltaire.com> <20090320134640.GG6689@sashak.voltaire.com> <20090320140029.GH6689@sashak.voltaire.com> Message-ID: <20090321012732.GB8262@sashak.voltaire.com> On 10:07 Fri 20 Mar , Hal Rosenstock wrote: > > It does eliminate the duplication in ib_mad_f but add additional calls > to mad_dump_perfcounters_xmt/rcv_sl. Is that really better ? There are no such calls, at least not now. Sasha From sashak at voltaire.com Fri Mar 20 18:28:11 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 21 Mar 2009 03:28:11 +0200 Subject: [ofa-general] OpenSM/IB management BOF at Sonoma In-Reply-To: <20090320172718.af93b06f.weiny2@llnl.gov> References: <20090320121358.GD6267@sashak.voltaire.com> <20090320100546.2f93de58.weiny2@llnl.gov> <20090321002532.GA8262@sashak.voltaire.com> <20090320172718.af93b06f.weiny2@llnl.gov> Message-ID: <20090321012811.GC8262@sashak.voltaire.com> On 17:27 Fri 20 Mar , Ira Weiny wrote: > If last year is any indication I think it will have to be after the presentations. > > Perhaps Tues night? I would be willing to skip some presentations perhaps even "OpenFabrics Alliance Annual Meeting" (Tues 5:30) to get started a bit earlier? > > Anyone else? Works fine for me. Sasha From sashak at voltaire.com Fri Mar 20 18:38:05 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 21 Mar 2009 03:38:05 +0200 Subject: [ofa-general] Couple of questions about OpenSM In-Reply-To: <49C3BE63.1060703@morey-chaisemartin.com> References: <49C21A7F.3090203@ext.bull.net> <20090320115941.GC6267@sashak.voltaire.com> <49C3BE63.1060703@morey-chaisemartin.com> Message-ID: <20090321013805.GE8262@sashak.voltaire.com> On 17:03 Fri 20 Mar , Nicolas Morey-Chaisemartin wrote: > > Good. There should be one table for each switch and each of them as one entry per (port/lid) couple no? One minhops table per switch which consists hops per port arrays for *only* switches' lids (CA and router lids entries are never used). > Well it about 15% of computation time (for Ftree at least) is spent in set_hops and specifically in malloc/memset part > If we do it only once at the start, it should probably be much faster ! > > By the way, is there a reason you don't use likely/unlikely commands in conditions? No any special reason. Sasha From sashak at voltaire.com Fri Mar 20 18:46:56 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 21 Mar 2009 03:46:56 +0200 Subject: [ofa-general] Re: [PATCH] ibim/ibsim.c Fixed ibsim console so it doesn't eat up all CPU when creading from a pipe In-Reply-To: <49C3BE71.205@morey-chaisemartin.com> References: <49C39315.5070802@ext.bull.net> <20090320132414.GC6689@sashak.voltaire.com> <49C3BE71.205@morey-chaisemartin.com> Message-ID: <20090321014656.GF8262@sashak.voltaire.com> On 17:04 Fri 20 Mar , Nicolas Morey-Chaisemartin wrote: > > I guess so but in which case would it fail? read() can fail for couple of reason (see ERRORS section of 'man 2 read'). EAGAIN and EINTR should not be a fatal for ibsim. Sasha From sashak at voltaire.com Fri Mar 20 18:51:46 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 21 Mar 2009 03:51:46 +0200 Subject: [ofa-general] Re: OpenSM build now requires lex (flex) and yacc (bison)? In-Reply-To: <20090320102319.5b2ed182.weiny2@llnl.gov> References: <20090312133949.2fe26c07.weiny2@llnl.gov> <20090320134307.GF6689@sashak.voltaire.com> <20090320102319.5b2ed182.weiny2@llnl.gov> Message-ID: <20090321015146.GG8262@sashak.voltaire.com> On 10:23 Fri 20 Mar , Ira Weiny wrote: > > On the other hand I have polled around here and no one knows of a rpm distro > which does not provide bison or flex. Perhaps some of the other distros will > chime in if this will break them. I remember at least story byacc instead bison preinstalled by default. Maybe I saw some other tools, but I'm not sure now. Hal, do you remember what distro it was? Sasha From sashak at voltaire.com Fri Mar 20 18:56:30 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Sat, 21 Mar 2009 03:56:30 +0200 Subject: [ofa-general] Re: [PATCH] infiniband-diags/perfquery.8: Fix typo in short option for PortXmitDataSL counters In-Reply-To: <20090320231222.GB1424@comcast.net> References: <20090320231222.GB1424@comcast.net> Message-ID: <20090321015630.GB8646@sashak.voltaire.com> On 18:12 Fri 20 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From shemminger at vyatta.com Fri Mar 20 22:35:30 2009 From: shemminger at vyatta.com (Stephen Hemminger) Date: Fri, 20 Mar 2009 22:35:30 -0700 Subject: [ofa-general] [PATCH 03/77] infiniband: convert c2 to net_device_ops References: <20090321053527.316395697@vyatta.com> Message-ID: <20090321053712.797092692@vyatta.com> An embedded and charset-unspecified text was scrubbed... Name: infiniband-c2.patch URL: From shemminger at vyatta.com Fri Mar 20 22:35:31 2009 From: shemminger at vyatta.com (Stephen Hemminger) Date: Fri, 20 Mar 2009 22:35:31 -0700 Subject: [ofa-general] [PATCH 04/77] infiniband: convert nes driver to net_device_ops References: <20090321053527.316395697@vyatta.com> Message-ID: <20090321053712.868239443@vyatta.com> An embedded and charset-unspecified text was scrubbed... Name: inifiband-nes.patch URL: From shemminger at vyatta.com Fri Mar 20 22:35:32 2009 From: shemminger at vyatta.com (Stephen Hemminger) Date: Fri, 20 Mar 2009 22:35:32 -0700 Subject: [ofa-general] [PATCH 05/77] infiniband: convert ipoib to net_device_ops References: <20090321053527.316395697@vyatta.com> Message-ID: <20090321053712.940955485@vyatta.com> An embedded and charset-unspecified text was scrubbed... Name: ipoib.patch URL: From devel at morey-chaisemartin.com Sat Mar 21 00:51:43 2009 From: devel at morey-chaisemartin.com (Nicolas Morey-Chaisemartin) Date: Sat, 21 Mar 2009 08:51:43 +0100 Subject: [ofa-general] Couple of questions about OpenSM In-Reply-To: <20090321013805.GE8262@sashak.voltaire.com> References: <49C21A7F.3090203@ext.bull.net> <20090320115941.GC6267@sashak.voltaire.com> <49C3BE63.1060703@morey-chaisemartin.com> <20090321013805.GE8262@sashak.voltaire.com> Message-ID: <49C49C8F.8000608@morey-chaisemartin.com> Sasha Khapyorsky a écrit : > On 17:03 Fri 20 Mar , Nicolas Morey-Chaisemartin wrote: >> Good. There should be one table for each switch and each of them as one entry per (port/lid) couple no? > > One minhops table per switch which consists hops per port arrays for > *only* switches' lids (CA and router lids entries are never used). > It seems to me that all the entries (wether it is CA, Switch, Router, etc..) are used in the Ftree algorithm. The only tests done is wether this is a reald LID or not. If I'm right, first test would be not to set hops if target lid is not a a switch which would already gain a some time (and memory). Nicolas From devel at morey-chaisemartin.com Sat Mar 21 01:00:29 2009 From: devel at morey-chaisemartin.com (Nicolas Morey-Chaisemartin) Date: Sat, 21 Mar 2009 09:00:29 +0100 Subject: [ofa-general] Couple of questions about OpenSM In-Reply-To: <49C49C8F.8000608@morey-chaisemartin.com> References: <49C21A7F.3090203@ext.bull.net> <20090320115941.GC6267@sashak.voltaire.com> <49C3BE63.1060703@morey-chaisemartin.com> <20090321013805.GE8262@sashak.voltaire.com> <49C49C8F.8000608@morey-chaisemartin.com> Message-ID: <49C49E9D.6080209@morey-chaisemartin.com> Nicolas Morey-Chaisemartin a écrit : > Sasha Khapyorsky a écrit : >> On 17:03 Fri 20 Mar , Nicolas Morey-Chaisemartin wrote: >>> Good. There should be one table for each switch and each of them as one entry per (port/lid) couple no? >> One minhops table per switch which consists hops per port arrays for >> *only* switches' lids (CA and router lids entries are never used). >> > > It seems to me that all the entries (wether it is CA, Switch, Router, etc..) are used in the Ftree algorithm. > The only tests done is wether this is a reald LID or not. > > If I'm right, first test would be not to set hops if target lid is not a a switch which would already gain a some time (and memory). > > Nicolas Wait, what I'm saying is stupid. Fat-Tree needs the hops table to be set even for CA lics to detect if we are coming from a shorter path or not... What we don't need however is to have it set for each port the remote portgroup, but just a global switch value. Nicolas From vlad at lists.openfabrics.org Sat Mar 21 03:24:33 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sat, 21 Mar 2009 03:24:33 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090321-0200 daily build status Message-ID: <20090321102433.54A6BE61076@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From hal.rosenstock at gmail.com Sat Mar 21 04:16:46 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Sat, 21 Mar 2009 07:16:46 -0400 Subject: [ofa-general] ***SPAM*** Re: OpenSM build now requires lex (flex) and yacc (bison)? In-Reply-To: <20090321015146.GG8262@sashak.voltaire.com> References: <20090312133949.2fe26c07.weiny2@llnl.gov> <20090320134307.GF6689@sashak.voltaire.com> <20090320102319.5b2ed182.weiny2@llnl.gov> <20090321015146.GG8262@sashak.voltaire.com> Message-ID: On Fri, Mar 20, 2009 at 9:51 PM, Sasha Khapyorsky wrote: > On 10:23 Fri 20 Mar     , Ira Weiny wrote: >> >> On the other hand I have polled around here and no one knows of a rpm distro >> which does not provide bison or flex.  Perhaps some of the other distros will >> chime in if this will break them. > > I remember at least story byacc instead bison preinstalled by default. > Maybe I saw some other tools, but I'm not sure now. > Hal, do you remember what distro it was? At least one of the free distros does this. -- Hal > Sasha From hal.rosenstock at gmail.com Sat Mar 21 04:21:09 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Sat, 21 Mar 2009 07:21:09 -0400 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] libibmad: Add PortSelect and CounterSelect fields for PortXmit/RcvDataSL attributes In-Reply-To: <20090321012732.GB8262@sashak.voltaire.com> References: <20090313203620.GA6173@comcast.net> <20090320133633.GE6689@sashak.voltaire.com> <20090320134640.GG6689@sashak.voltaire.com> <20090320140029.GH6689@sashak.voltaire.com> <20090321012732.GB8262@sashak.voltaire.com> Message-ID: On Fri, Mar 20, 2009 at 9:27 PM, Sasha Khapyorsky wrote: > On 10:07 Fri 20 Mar     , Hal Rosenstock wrote: >> >> It does eliminate the duplication in ib_mad_f but add additional calls >> to mad_dump_perfcounters_xmt/rcv_sl. Is that really better ? > > There are no such calls, at least not now. Huh ? They already exist in libibmad/src/dump.c, added by Or. Anyhow, the difference is minor and it's six of one half dozen of other and you have both approaches in patches so you can decide. -- Hal > Sasha > From hnrose at comcast.net Sat Mar 21 05:08:40 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Sat, 21 Mar 2009 07:08:40 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] libibmad/sa.c: Cosmetic formatting changes Message-ID: <20090321120840.GA10982@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/libibmad/src/sa.c b/libibmad/src/sa.c index 1a5740b..7bd094e 100644 --- a/libibmad/src/sa.c +++ b/libibmad/src/sa.c @@ -45,8 +45,8 @@ #undef DEBUG #define DEBUG if (ibdebug) IBWARN -uint8_t *sa_rpc_call(const struct ibmad_port *ibmad_port, void *rcvbuf, ib_portid_t * portid, - ib_sa_call_t * sa, unsigned timeout) +uint8_t *sa_rpc_call(const struct ibmad_port *ibmad_port, void *rcvbuf, + ib_portid_t * portid, ib_sa_call_t * sa, unsigned timeout) { ib_rpc_t rpc = { 0 }; uint8_t *p; @@ -73,7 +73,7 @@ uint8_t *sa_rpc_call(const struct ibmad_port *ibmad_port, void *rcvbuf, ib_porti if (!portid->qkey) portid->qkey = IB_DEFAULT_QP1_QKEY; - p = mad_rpc_rmpp(ibmad_port, &rpc, portid, 0 /*&sa->rmpp */ , rcvbuf); /* TODO: RMPP */ + p = mad_rpc_rmpp(ibmad_port, &rpc, portid, 0 /*&sa->rmpp */, rcvbuf); /* TODO: RMPP */ sa->recsz = rpc.recsz; From swise at opengridcomputing.com Sat Mar 21 11:26:22 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Sat, 21 Mar 2009 13:26:22 -0500 Subject: [ofa-general] Re: [PATCH 03/77] infiniband: convert c2 to net_device_ops In-Reply-To: <20090321053712.797092692@vyatta.com> References: <20090321053527.316395697@vyatta.com> <20090321053712.797092692@vyatta.com> Message-ID: <49C5314E.8010507@opengridcomputing.com> Reviewed-by: Steve Wise From davem at davemloft.net Sat Mar 21 19:34:17 2009 From: davem at davemloft.net (David Miller) Date: Sat, 21 Mar 2009 19:34:17 -0700 (PDT) Subject: [ofa-general] Re: [PATCH 03/77] infiniband: convert c2 to net_device_ops In-Reply-To: <20090321053712.797092692@vyatta.com> References: <20090321053527.316395697@vyatta.com> <20090321053712.797092692@vyatta.com> Message-ID: <20090321.193417.178915231.davem@davemloft.net> From: Stephen Hemminger Date: Fri, 20 Mar 2009 22:35:30 -0700 > Convert this driver to new net_device_ops infrastructure. > Also use default net_device get-stats infrastructure > > Signed-off-by: Stephen Hemminger Applied. From davem at davemloft.net Sat Mar 21 19:34:20 2009 From: davem at davemloft.net (David Miller) Date: Sat, 21 Mar 2009 19:34:20 -0700 (PDT) Subject: [ofa-general] Re: [PATCH 04/77] infiniband: convert nes driver to net_device_ops In-Reply-To: <20090321053712.868239443@vyatta.com> References: <20090321053527.316395697@vyatta.com> <20090321053712.868239443@vyatta.com> Message-ID: <20090321.193420.211698588.davem@davemloft.net> From: Stephen Hemminger Date: Fri, 20 Mar 2009 22:35:31 -0700 > Also, removed unnecessary memset() sinc alloc_netdev returns > zeroed memory. > > Signed-off-by: Stephen Hemminger Applied. From davem at davemloft.net Sat Mar 21 19:34:21 2009 From: davem at davemloft.net (David Miller) Date: Sat, 21 Mar 2009 19:34:21 -0700 (PDT) Subject: [ofa-general] Re: [PATCH 05/77] infiniband: convert ipoib to net_device_ops In-Reply-To: <20090321053712.940955485@vyatta.com> References: <20090321053527.316395697@vyatta.com> <20090321053712.940955485@vyatta.com> Message-ID: <20090321.193421.174174760.davem@davemloft.net> From: Stephen Hemminger Date: Fri, 20 Mar 2009 22:35:32 -0700 > Signed-off-by: Stephen Hemminger Applied. From ogerlitz at voltaire.com Sat Mar 21 23:53:30 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 22 Mar 2009 08:53:30 +0200 Subject: [ofa-general] [PATCH] ib-diag/vendstat: counter-group-info & config-counter-group vendor mads In-Reply-To: References: Message-ID: <49C5E06A.6030401@voltaire.com> Hal Rosenstock wrote: > Do these changes persist across an IS4 reboot? > I don't know. > would you also submit a man page update for this? > OK, will do. Or. From itaib at mellanox.co.il Sun Mar 22 02:14:01 2009 From: itaib at mellanox.co.il (Itai Baz) Date: Sun, 22 Mar 2009 11:14:01 +0200 Subject: [ofa-general] RE: [PATCH] libibmad: added support for handling of BM (Baseboard management) MADs - FIXED without rmpp In-Reply-To: References: <200903191625.24559.itaib@mellanox.com> <5D49E7A8952DC44FB38C38FA0D758EAD021D0EDD@mtlexch01.mtl.com> Message-ID: <5D49E7A8952DC44FB38C38FA0D758EAD021D1704@mtlexch01.mtl.com> As far as I know and checked, there is no need to implement a special transactions mechanism, and the code is fine There is transaction mechanism defined for IB-ML and IB-MME in the spec, not for BM MADs. Itai -----Original Message----- From: Hal Rosenstock [mailto:hal.rosenstock at gmail.com] Sent: Thursday, March 19, 2009 5:49 PM To: Itai Baz Cc: Sasha Khapyorsky; OpenIB Subject: Re: [PATCH] libibmad: added support for handling of BM (Baseboard management) MADs - FIXED without rmpp On Thu, Mar 19, 2009 at 11:00 AM, Itai Baz wrote: > As far as I'm aware, the response_expected handling is sufficient. > BMGet and BMSet always have response. > BMSend may be used both as a request from client to server, and as a > response from server to client Yes, but is transaction handling (request/response) used or needed on some BMSends ? -- Hal > -----Original Message----- > From: Hal Rosenstock [mailto:hal.rosenstock at gmail.com] > Sent: Thursday, March 19, 2009 4:50 PM > To: Itai Baz > Cc: Sasha Khapyorsky; OpenIB > Subject: Re: [PATCH] libibmad: added support for handling of BM > (Baseboard management) MADs - FIXED without rmpp > > On Thu, Mar 19, 2009 at 10:25 AM, Itai Baz wrote: >> This patch adds support for handling of BM (Baseboard management) > MADs. >> >> I checked Hal's comment regarding RMPP, indeed there is no need for >> it > >> for BM, so I have removed rmpp, and i'm using now mad_rpc > > Thanks. > > Also, is the response_expected handling sufficient ? For any BMSend, > it will say no response expected. Is that what is really desired ? I'm > not sure on what basis the BMA sets the response bit in the AM on a > "response" to a BMSend but that is used in the kernel (to determine an > incoming BM response). Should this mimic that idea ? > > -- Hal > > > From vlad at lists.openfabrics.org Sun Mar 22 03:24:49 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sun, 22 Mar 2009 03:24:49 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090322-0200 daily build status Message-ID: <20090322102450.04859E6106F@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From rdreier at cisco.com Sun Mar 22 09:12:57 2009 From: rdreier at cisco.com (Roland Dreier) Date: Sun, 22 Mar 2009 09:12:57 -0700 Subject: [ofa-general] [PATCH 03/77] infiniband: convert c2 to net_device_ops In-Reply-To: <20090321053712.797092692@vyatta.com> (Stephen Hemminger's message of "Fri, 20 Mar 2009 22:35:30 -0700") References: <20090321053527.316395697@vyatta.com> <20090321053712.797092692@vyatta.com> Message-ID: Don't I already have this queued in my for-next branch? Or is this an updated version? [And same applies to the nes driver changes] - R. From jlentini at netapp.com Sun Mar 22 11:27:03 2009 From: jlentini at netapp.com (James Lentini) Date: Sun, 22 Mar 2009 14:27:03 -0400 (EDT) Subject: [ofa-general][PATCH] NFS-RDMA fix for connectX fast registration wr, In-Reply-To: <49C422FC.7040303@mellanox.com> References: <49C422FC.7040303@mellanox.com> Message-ID: On Fri, 20 Mar 2009, Vu Pham wrote: > Hi Jon, > > Here is the patch for the file > kernel_patches/fixes/nfsrdma_to_2_6_28_18.patch. This patch will fix the fast > registration wr for Mellanox ConnectX. > > ConnectX FRMR requires local write enable together with remote rdma write > enable > > thanks, > -vu > Vu, Are you going to submit this fix upstream? -james > --- ofa_kernel-1.4.1.config/kernel_patches/fixes/nfsrdma_to_2.6.28_13.patch 2009-03-19 08:42:07.000000000 -0700 > +++ ofa_kernel-1.4.1/kernel_patches/fixes/nfsrdma_to_2.6.28_13.patch 2009-03-19 08:49:22.000000000 -0700 > @@ -167,7 +167,7 @@ > case RPCRDMA_MTHCAFMR: > case RPCRDMA_MEMWINDOWS_ASYNC: > case RPCRDMA_MEMWINDOWS: > -@@ -1391,6 +1457,96 @@ rpcrdma_unmap_one(struct rpcrdma_ia *ia, struct rpcrdma_mr_seg *seg) > +@@ -1391,6 +1457,98 @@ rpcrdma_unmap_one(struct rpcrdma_ia *ia, struct rpcrdma_mr_seg *seg) > } > > static int > @@ -214,8 +214,10 @@ > + frmr_wr.wr.fast_reg.page_list_len = i; > + frmr_wr.wr.fast_reg.page_shift = PAGE_SHIFT; > + frmr_wr.wr.fast_reg.length = i << PAGE_SHIFT; > -+ frmr_wr.wr.fast_reg.access_flags = (writing ? > -+ IB_ACCESS_REMOTE_WRITE : IB_ACCESS_REMOTE_READ); > ++ frmr_wr.wr.fast_reg.access_flags = > ++ (writing ? > ++ (IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE) : > ++ IB_ACCESS_REMOTE_READ); > + frmr_wr.wr.fast_reg.rkey = seg1->mr_chunk.rl_mw->r.frmr.fr_mr->rkey; > + DECR_CQCOUNT(&r_xprt->rx_ep); > + From tmtalpey at gmail.com Sun Mar 22 15:09:11 2009 From: tmtalpey at gmail.com (Tom Talpey) Date: Sun, 22 Mar 2009 18:09:11 -0400 Subject: ***SPAM*** Re: [ofa-general][PATCH] NFS-RDMA fix for connectX fast registration wr, In-Reply-To: References: <49C422FC.7040303@mellanox.com> Message-ID: <49c6b719.0e538c0a.3e06.ffff8ed8@mx.google.com> At 02:27 PM 3/22/2009, James Lentini wrote: > > >On Fri, 20 Mar 2009, Vu Pham wrote: > >> Hi Jon, >> >> Here is the patch for the file >> kernel_patches/fixes/nfsrdma_to_2_6_28_18.patch. This patch will fix the fast >> registration wr for Mellanox ConnectX. >> >> ConnectX FRMR requires local write enable together with remote rdma write >> enable >> >> thanks, >> -vu >> > > >Vu, > >Are you going to submit this fix upstream? Agreed - if setting this bit fixes the FRMR issue, then it should go to Trond asap for inclusion in 2.6.30 (hopefully). Can someone verify that it works? I don't have access to ConnectX hardware as I am at the OFA workshop. Tom. > >-james > > >> --- >ofa_kernel-1.4.1.config/kernel_patches/fixes/nfsrdma_to_2.6.28_13.patc >h 2009-03-19 08:42:07.000000000 -0700 >> +++ ofa_kernel-1.4.1/kernel_patches/fixes/nfsrdma_to_2.6.28_13.patch >2009-03-19 08:49:22.000000000 -0700 >> @@ -167,7 +167,7 @@ >> case RPCRDMA_MTHCAFMR: >> case RPCRDMA_MEMWINDOWS_ASYNC: >> case RPCRDMA_MEMWINDOWS: >> -@@ -1391,6 +1457,96 @@ rpcrdma_unmap_one(struct rpcrdma_ia *ia, >struct rpcrdma_mr_seg *seg) >> +@@ -1391,6 +1457,98 @@ rpcrdma_unmap_one(struct rpcrdma_ia *ia, >struct rpcrdma_mr_seg *seg) >> } >> >> static int >> @@ -214,8 +214,10 @@ >> + frmr_wr.wr.fast_reg.page_list_len = i; >> + frmr_wr.wr.fast_reg.page_shift = PAGE_SHIFT; >> + frmr_wr.wr.fast_reg.length = i << PAGE_SHIFT; >> -+ frmr_wr.wr.fast_reg.access_flags = (writing ? >> -+ IB_ACCESS_REMOTE_WRITE : IB_ACCESS_REMOTE_READ); >> ++ frmr_wr.wr.fast_reg.access_flags = >> ++ (writing ? >> ++ (IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE) : >> ++ IB_ACCESS_REMOTE_READ); >> + frmr_wr.wr.fast_reg.rkey = seg1->mr_chunk.rl_mw->r.frmr.fr_mr->rkey; >> + DECR_CQCOUNT(&r_xprt->rx_ep); >> + From davem at davemloft.net Sun Mar 22 21:17:04 2009 From: davem at davemloft.net (David Miller) Date: Sun, 22 Mar 2009 21:17:04 -0700 (PDT) Subject: [ofa-general] [PATCH 03/77] infiniband: convert c2 to net_device_ops In-Reply-To: References: <20090321053527.316395697@vyatta.com> <20090321053712.797092692@vyatta.com> Message-ID: <20090322.211704.141753947.davem@davemloft.net> From: Roland Dreier Date: Sun, 22 Mar 2009 09:12:57 -0700 > Don't I already have this queued in my for-next branch? > > Or is this an updated version? > > [And same applies to the nes driver changes] Any chance you can drop the copies in your tree? The reason I'm asking is that once all the conversions are in net-next-2.6 which hit the i386 allmodconfig build there is a change we can add (which in fast is in net-next-2.6 now) which exposes a config option to turn all of the pre-netdev_ops compatibility stuff off. We can't do that until i386 builds in all configurations. From poknam at gmail.com Sun Mar 22 23:37:05 2009 From: poknam at gmail.com (PN) Date: Mon, 23 Mar 2009 14:37:05 +0800 Subject: [ofa-general] ***SPAM*** mpitests_mvapich2_intel rpm problem Message-ID: <92daa7bf0903222337m514c40a8ideb687b67774f49c@mail.gmail.com> I have another problem on installing the OFED-1.4.1-rc2 on CentOS 5.2. I can build all the rpm, but there is an error while installing the mpitests_mvapich2_intel-3.1-891.x86_64.rpm .... Install mpitests_mvapich_gcc RPM: Running rpm -iv /root/OFED-1.4.1-rc2/OFED-1.4.1-rc2/RPMS/centos-release-5-2.el5.centos/x86_64/mpitests_mvapich_gcc-3.1-891.x86_64.rpm Install mpitests_mvapich_intel RPM: Running rpm -iv /root/OFED-1.4.1-rc2/OFED-1.4.1-rc2/RPMS/centos-release-5-2.el5.centos/x86_64/mpitests_mvapich_intel-3.1-891.x86_64.rpm Install mpitests_mvapich2_gcc RPM: Running rpm -iv /root/OFED-1.4.1-rc2/OFED-1.4.1-rc2/RPMS/centos-release-5-2.el5.centos/x86_64/mpitests_mvapich2_gcc-3.1-891.x86_64.rpm Install mpitests_mvapich2_intel RPM: Running rpm -iv /root/OFED-1.4.1-rc2/OFED-1.4.1-rc2/RPMS/centos-release-5-2.el5.centos/x86_64/mpitests_mvapich2_intel-3.1-891.x86_64.rpm Failed to install mpitests_mvapich2_intel RPM See /tmp/OFED.9482.logs/mpitests_mvapich2_intel.rpminstall.log # cat /tmp/OFED.9482.logs/mpitests_mvapich2_intel.rpminstall.log error: Failed dependencies: libimf.so()(64bit) is needed by mpitests_mvapich2_intel-3.1-891.x86_64 libintlc.so.5()(64bit) is needed by mpitests_mvapich2_intel-3.1-891.x86_64 libsvml.so()(64bit) is needed by mpitests_mvapich2_intel-3.1-891.x86_64 # echo $LD_LIBRARY_PATH /opt/intel/Compiler/11.0/081/lib/intel64:/opt/intel/Compiler/11.0/081/ipp/em64t/sharedlib:/opt/intel/Compiler/11.0/081/mkl/lib/em64t:/opt/intel/Compiler/11.0/081/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib:/opt/intel/Compiler/11.0/081/lib/intel64:/opt/intel/Compiler/11.0/081/ipp/em64t/sharedlib:/opt/intel/Compiler/11.0/081/mkl/lib/em64t:/opt/intel/Compiler/11.0/081/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib I've installed Intel Compiler 11.0.081 and I found that all the required .so are located in /opt/intel/Compiler/11.0/081/lib/intel64 and the location is already in LD_LIBRARY_PATH, but somehow the rpm cannot located the library files. I found that the problem also occurred in OFED-1.4.1-rc1 and OFED-1.4.1. Any idea on how to solve this? Thanks in advance. Regards, PN -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlad at lists.openfabrics.org Mon Mar 23 03:21:33 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Mon, 23 Mar 2009 03:21:33 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090323-0200 daily build status Message-ID: <20090323102134.289E8E60810@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From hnrose at comcast.net Mon Mar 23 05:28:20 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Mon, 23 Mar 2009 07:28:20 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] opensm/include/ib_types.h: Fix some typos Message-ID: <20090323122820.GA19210@comcast.net> Also, some cosmetic formatting changes Signed-off-by: Hal Rosenstock --- diff --git a/opensm/include/iba/ib_types.h b/opensm/include/iba/ib_types.h index eac85f3..32b916e 100644 --- a/opensm/include/iba/ib_types.h +++ b/opensm/include/iba/ib_types.h @@ -3295,10 +3295,10 @@ typedef struct _ib_class_port_info { * Maximum expected response time and additional * supported capabilities of this management class. * -* redr_gid +* redir_gid * GID to use for redirection, or zero * -* recdir_tc_sl_fl +* redir_tc_sl_fl * Traffic class, service level and flow label the requester * should use if the service is redirected. * @@ -5933,11 +5933,11 @@ ib_switch_info_clear_state_change(IN ib_switch_info_t * const p_si) /* * PARAMETERS -* p_ni -* [in] Pointer to a PortInfo attribute. +* p_si +* [in] Pointer to a SwitchInfo attribute. * * RETURN VALUES -* Returns the LMC value assigned to this port. +* None * * NOTES * @@ -8406,8 +8406,8 @@ typedef enum _ib_async_event_t { * alternate path. * * IB_AE_LOCAL_FATAL -* A catastrophic HCA error occurred which cannot be attributed to any -* resource; behavior is indeterminate. +* A catastrophic HCA error occurred which cannot be attributed to +* any resource; behavior is indeterminate. * * IB_AE_PKEY_TRAP * A PKEY violation was detected. This event is optional. @@ -8416,15 +8416,15 @@ typedef enum _ib_async_event_t { * A QKEY violation was detected. This event is optional. * * IB_AE_MKEY_TRAP -* An MKEY violation was detected. This event is optional. +* A MKEY violation was detected. This event is optional. * * IB_AE_PORT_TRAP * A port capability change was detected. This event is optional. * * IB_AE_SYSIMG_GUID_TRAP -* If the system image GUID is supported, this event indicates that the -* system image GUID of this HCA has been changed. This event is -* optional. +* If the system image GUID is supported, this event indicates that +* the system image GUID of this HCA has been changed. This event +* is optional. * * IB_AE_BUF_OVERRUN * The number of consecutive flow control update periods with at least From hnrose at comcast.net Mon Mar 23 05:29:22 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Mon, 23 Mar 2009 07:29:22 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] opensm: Some cosmetic changes Message-ID: <20090323122922.GB19210@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_pkey_rcv.c b/opensm/opensm/osm_pkey_rcv.c index 84d8ce1..e8e030e 100644 --- a/opensm/opensm/osm_pkey_rcv.c +++ b/opensm/opensm/osm_pkey_rcv.c @@ -51,7 +51,7 @@ /********************************************************************** **********************************************************************/ /* - * WE MIGHT ONLY RECEIVE GET or SET responses + * WE ONLY RECEIVE GET or SET responses */ void osm_pkey_rcv_process(IN void *context, IN void *data) { @@ -112,8 +112,8 @@ void osm_pkey_rcv_process(IN void *context, IN void *data) } /* - We do not mind if this is a result of a set or get - all we want is to - update the subnet. + We do not care if this is a result of a set or get - + all we want is to update the subnet. */ OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, "Got GetResp(PKey) block:%u port_num %u with GUID 0x%" diff --git a/opensm/opensm/osm_slvl_map_rcv.c b/opensm/opensm/osm_slvl_map_rcv.c index b3f0a4c..f5a7370 100644 --- a/opensm/opensm/osm_slvl_map_rcv.c +++ b/opensm/opensm/osm_slvl_map_rcv.c @@ -58,7 +58,7 @@ /********************************************************************** **********************************************************************/ /* - * WE MIGHT ONLY RECEIVE A GET or SET responses + * WE ONLY RECEIVE GET or SET responses */ void osm_slvl_rcv_process(IN void *context, IN void *p_data) { @@ -120,8 +120,8 @@ void osm_slvl_rcv_process(IN void *context, IN void *p_data) } /* - We do not mind if this is a result of a set or get - all we want is to update - the subnet. + We do not care if this is a result of a set or get - + all we want is to update the subnet. */ OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, "Got SLtoVL get response in_port_num %u out_port_num %u with " diff --git a/opensm/opensm/osm_vl_arb_rcv.c b/opensm/opensm/osm_vl_arb_rcv.c index 89cf7b2..65adaf5 100644 --- a/opensm/opensm/osm_vl_arb_rcv.c +++ b/opensm/opensm/osm_vl_arb_rcv.c @@ -58,7 +58,7 @@ /********************************************************************** **********************************************************************/ /* - * WE MIGHT ONLY RECEIVE GET or SET responses + * WE ONLY RECEIVE GET or SET responses */ void osm_vla_rcv_process(IN void *context, IN void *data) { @@ -117,8 +117,8 @@ void osm_vla_rcv_process(IN void *context, IN void *data) } /* - We do not mind if this is a result of a set or get - all we want is to update - the subnet. + We do not care if this is a result of a set or get - + all we want is to update the subnet. */ OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, "Got GetResp(VLArb) block:%u port_num %u with GUID 0x%" From hnrose at comcast.net Mon Mar 23 05:30:12 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Mon, 23 Mar 2009 07:30:12 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] opensm: Remove __osm_ prefixes Message-ID: <20090323123012.GC19210@comcast.net> Also, some cosmetic formatting changes Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_db_files.c b/opensm/opensm/osm_db_files.c index 907b7c7..12457f4 100644 --- a/opensm/opensm/osm_db_files.c +++ b/opensm/opensm/osm_db_files.c @@ -423,7 +423,7 @@ Exit: /*************************************************************************** ***************************************************************************/ -static int __osm_dump_tbl_entry(st_data_t key, st_data_t val, st_data_t arg) +static int dump_tbl_entry(st_data_t key, st_data_t val, st_data_t arg) { FILE *p_file = (FILE *) arg; char *p_key = (char *)key; @@ -462,8 +462,7 @@ int osm_db_store(IN osm_db_domain_t * p_domain) goto Exit; } - st_foreach(p_domain_imp->p_hash, __osm_dump_tbl_entry, - (st_data_t) p_file); + st_foreach(p_domain_imp->p_hash, dump_tbl_entry, (st_data_t) p_file); fclose(p_file); /* move the domain file */ @@ -491,7 +490,7 @@ Exit: ***************************************************************************/ /* simply de-allocate the key and the value and return the code that makes the st_foreach delete the entry */ -static int __osm_clear_tbl_entry(st_data_t key, st_data_t val, st_data_t arg) +static int clear_tbl_entry(st_data_t key, st_data_t val, st_data_t arg) { free((char *)key); free((char *)val); @@ -504,8 +503,7 @@ int osm_db_clear(IN osm_db_domain_t * p_domain) (osm_db_domain_imp_t *) p_domain->p_domain_imp; cl_spinlock_acquire(&p_domain_imp->lock); - st_foreach(p_domain_imp->p_hash, __osm_clear_tbl_entry, - (st_data_t) NULL); + st_foreach(p_domain_imp->p_hash, clear_tbl_entry, (st_data_t) NULL); cl_spinlock_release(&p_domain_imp->lock); return 0; @@ -513,7 +511,7 @@ int osm_db_clear(IN osm_db_domain_t * p_domain) /*************************************************************************** ***************************************************************************/ -static int __osm_get_key_of_tbl_entry(st_data_t key, st_data_t val, +static int get_key_of_tbl_entry(st_data_t key, st_data_t val, st_data_t arg) { cl_list_t *p_list = (cl_list_t *) arg; @@ -528,8 +526,8 @@ int osm_db_keys(IN osm_db_domain_t * p_domain, OUT cl_list_t * p_key_list) cl_spinlock_acquire(&p_domain_imp->lock); - st_foreach(p_domain_imp->p_hash, - __osm_get_key_of_tbl_entry, (st_data_t) p_key_list); + st_foreach(p_domain_imp->p_hash, get_key_of_tbl_entry, + (st_data_t) p_key_list); cl_spinlock_release(&p_domain_imp->lock); diff --git a/opensm/opensm/osm_db_pack.c b/opensm/opensm/osm_db_pack.c index bf56169..a29bdc6 100644 --- a/opensm/opensm/osm_db_pack.c +++ b/opensm/opensm/osm_db_pack.c @@ -41,24 +41,24 @@ #include #include -static inline void __osm_pack_guid(uint64_t guid, char *p_guid_str) +static inline void pack_guid(uint64_t guid, char *p_guid_str) { sprintf(p_guid_str, "0x%016" PRIx64, guid); } -static inline uint64_t __osm_unpack_guid(char *p_guid_str) +static inline uint64_t unpack_guid(char *p_guid_str) { return strtoull(p_guid_str, NULL, 0); } static inline void -__osm_pack_lids(uint16_t min_lid, uint16_t max_lid, char *p_lid_str) +pack_lids(uint16_t min_lid, uint16_t max_lid, char *p_lid_str) { sprintf(p_lid_str, "0x%04x 0x%04x", min_lid, max_lid); } static inline int -__osm_unpack_lids(IN char *p_lid_str, +unpack_lids(IN char *p_lid_str, OUT uint16_t * p_min_lid, OUT uint16_t * p_max_lid) { unsigned long tmp; @@ -104,7 +104,7 @@ osm_db_guid2lid_guids(IN osm_db_domain_t * const p_g2l, (osm_db_guid_elem_t *) malloc(sizeof(osm_db_guid_elem_t)); CL_ASSERT(p_guid_elem != NULL); - p_guid_elem->guid = __osm_unpack_guid(p_key); + p_guid_elem->guid = unpack_guid(p_key); cl_qlist_insert_head(p_guid_list, &p_guid_elem->item); } @@ -121,11 +121,11 @@ osm_db_guid2lid_get(IN osm_db_domain_t * const p_g2l, char *p_lid_str; uint16_t min_lid, max_lid; - __osm_pack_guid(guid, guid_str); + pack_guid(guid, guid_str); p_lid_str = osm_db_lookup(p_g2l, guid_str); if (!p_lid_str) return 1; - if (__osm_unpack_lids(p_lid_str, &min_lid, &max_lid)) + if (unpack_lids(p_lid_str, &min_lid, &max_lid)) return 1; if (p_min_lid) @@ -143,8 +143,8 @@ osm_db_guid2lid_set(IN osm_db_domain_t * const p_g2l, char guid_str[20]; char lid_str[16]; - __osm_pack_guid(guid, guid_str); - __osm_pack_lids(min_lid, max_lid, lid_str); + pack_guid(guid, guid_str); + pack_lids(min_lid, max_lid, lid_str); return (osm_db_update(p_g2l, guid_str, lid_str)); } @@ -152,6 +152,6 @@ osm_db_guid2lid_set(IN osm_db_domain_t * const p_g2l, int osm_db_guid2lid_delete(IN osm_db_domain_t * const p_g2l, IN uint64_t guid) { char guid_str[20]; - __osm_pack_guid(guid, guid_str); + pack_guid(guid, guid_str); return (osm_db_delete(p_g2l, guid_str)); } diff --git a/opensm/opensm/osm_drop_mgr.c b/opensm/opensm/osm_drop_mgr.c index 215a155..f994882 100644 --- a/opensm/opensm/osm_drop_mgr.c +++ b/opensm/opensm/osm_drop_mgr.c @@ -66,7 +66,7 @@ /********************************************************************** **********************************************************************/ static void -__osm_drop_mgr_remove_router(osm_sm_t * sm, IN const ib_net64_t portguid) +drop_mgr_remove_router(osm_sm_t * sm, IN const ib_net64_t portguid) { osm_router_t *p_rtr; cl_qmap_t *p_rtr_guid_tbl; @@ -153,7 +153,7 @@ static void drop_mgr_clean_physp(osm_sm_t * sm, IN osm_physp_t * p_physp) /********************************************************************** **********************************************************************/ -static void __osm_drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) +static void drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) { ib_net64_t port_guid; osm_port_t *p_port_check; @@ -197,7 +197,7 @@ static void __osm_drop_mgr_remove_port(osm_sm_t * sm, IN osm_port_t * p_port) free(p_sm); } - __osm_drop_mgr_remove_router(sm, port_guid); + drop_mgr_remove_router(sm, port_guid); osm_port_get_lid_range_ho(p_port, &min_lid_ho, &max_lid_ho); @@ -270,7 +270,7 @@ Exit: /********************************************************************** **********************************************************************/ -static void __osm_drop_mgr_remove_switch(osm_sm_t * sm, IN osm_node_t * p_node) +static void drop_mgr_remove_switch(osm_sm_t * sm, IN osm_node_t * p_node) { osm_switch_t *p_sw; cl_qmap_t *p_sw_guid_tbl; @@ -297,7 +297,7 @@ static void __osm_drop_mgr_remove_switch(osm_sm_t * sm, IN osm_node_t * p_node) /********************************************************************** **********************************************************************/ static boolean_t -__osm_drop_mgr_process_node(osm_sm_t * sm, IN osm_node_t * p_node) +drop_mgr_process_node(osm_sm_t * sm, IN osm_node_t * p_node) { osm_physp_t *p_physp; osm_port_t *p_port; @@ -329,7 +329,7 @@ __osm_drop_mgr_process_node(osm_sm_t * sm, IN osm_node_t * p_node) p_port = osm_get_port_by_guid(sm->p_subn, port_guid); if (p_port) - __osm_drop_mgr_remove_port(sm, p_port); + drop_mgr_remove_port(sm, p_port); else drop_mgr_clean_physp(sm, p_physp); } @@ -338,7 +338,7 @@ __osm_drop_mgr_process_node(osm_sm_t * sm, IN osm_node_t * p_node) return_val = TRUE; if (p_node->sw) - __osm_drop_mgr_remove_switch(sm, p_node); + drop_mgr_remove_switch(sm, p_node); p_node_check = (osm_node_t *) cl_qmap_remove(&sm->p_subn->node_guid_tbl, @@ -358,7 +358,7 @@ __osm_drop_mgr_process_node(osm_sm_t * sm, IN osm_node_t * p_node) /********************************************************************** **********************************************************************/ -static void __osm_drop_mgr_check_node(osm_sm_t * sm, IN osm_node_t * p_node) +static void drop_mgr_check_node(osm_sm_t * sm, IN osm_node_t * p_node) { ib_net64_t node_guid; osm_physp_t *p_physp; @@ -383,7 +383,7 @@ static void __osm_drop_mgr_check_node(osm_sm_t * sm, IN osm_node_t * p_node) "Node 0x%016" PRIx64 " no switch in table\n", cl_ntoh64(node_guid)); - __osm_drop_mgr_process_node(sm, p_node); + drop_mgr_process_node(sm, p_node); goto Exit; } @@ -394,7 +394,7 @@ static void __osm_drop_mgr_check_node(osm_sm_t * sm, IN osm_node_t * p_node) "Node 0x%016" PRIx64 " no valid physical port 0\n", cl_ntoh64(node_guid)); - __osm_drop_mgr_process_node(sm, p_node); + drop_mgr_process_node(sm, p_node); goto Exit; } @@ -407,7 +407,7 @@ static void __osm_drop_mgr_check_node(osm_sm_t * sm, IN osm_node_t * p_node) "Node 0x%016" PRIx64 " has no port object\n", cl_ntoh64(node_guid)); - __osm_drop_mgr_process_node(sm, p_node); + drop_mgr_process_node(sm, p_node); goto Exit; } @@ -416,7 +416,7 @@ static void __osm_drop_mgr_check_node(osm_sm_t * sm, IN osm_node_t * p_node) "Node 0x%016" PRIx64 " port has discovery count zero\n", cl_ntoh64(node_guid)); - __osm_drop_mgr_process_node(sm, p_node); + drop_mgr_process_node(sm, p_node); goto Exit; } @@ -464,7 +464,7 @@ void osm_drop_mgr_process(osm_sm_t * sm) should therefore be removed from the subnet object. */ if (p_node->discovery_count == 0) - __osm_drop_mgr_process_node(sm, p_node); + drop_mgr_process_node(sm, p_node); } /* @@ -488,7 +488,7 @@ void osm_drop_mgr_process(osm_sm_t * sm) continue; /* We are handling a switch node */ - __osm_drop_mgr_check_node(sm, p_node); + drop_mgr_check_node(sm, p_node); } p_next_port = (osm_port_t *) cl_qmap_head(p_port_guid_tbl); @@ -508,7 +508,7 @@ void osm_drop_mgr_process(osm_sm_t * sm) If the port is unreachable, remove it from the guid table. */ if (p_port->discovery_count == 0) - __osm_drop_mgr_remove_port(sm, p_port); + drop_mgr_remove_port(sm, p_port); } CL_PLOCK_RELEASE(sm->p_lock); diff --git a/opensm/opensm/osm_dump.c b/opensm/opensm/osm_dump.c index 7e0ba47..865899d 100644 --- a/opensm/opensm/osm_dump.c +++ b/opensm/opensm/osm_dump.c @@ -130,7 +130,7 @@ static void dump_ucast_routes(cl_map_item_t *p_map_item, FILE *file, void *cxt) max_lid_ho = p_sw->max_lid_ho; - fprintf(file, "__osm_ucast_mgr_dump_ucast_routes: " + fprintf(file, "ucast_mgr_dump_ucast_routes: " "Switch 0x%016" PRIx64 "\nLID : Port : Hops : Optimal\n", cl_ntoh64(osm_node_get_node_guid(p_node))); diff --git a/opensm/opensm/osm_inform.c b/opensm/opensm/osm_inform.c index dd3f41e..6cc2923 100644 --- a/opensm/opensm/osm_inform.c +++ b/opensm/opensm/osm_inform.c @@ -296,8 +296,8 @@ osm_infr_remove_from_db(IN osm_subn_t * p_subn, * Given a target address to send to and the notice. * We need to send SubnAdmReport **********************************************************************/ -static ib_api_status_t __osm_send_report(IN osm_infr_t * p_infr_rec, /* the informinfo */ - IN ib_mad_notice_attr_t * p_ntc /* notice to send */ +static ib_api_status_t send_report(IN osm_infr_t * p_infr_rec, /* the informinfo */ + IN ib_mad_notice_attr_t * p_ntc /* notice to send */ ) { osm_madw_t *p_report_madw; @@ -524,7 +524,7 @@ __match_notice_to_inf_rec(IN cl_list_item_t * const p_list_item, /* send the report to the address provided in the inform record */ OSM_LOG(p_log, OSM_LOG_DEBUG, "MATCH! Sending Report...\n"); - __osm_send_report(p_infr_rec, p_ntc); + send_report(p_infr_rec, p_ntc); status = CL_SUCCESS; Exit: diff --git a/opensm/opensm/osm_lid_mgr.c b/opensm/opensm/osm_lid_mgr.c index e527337..ed384ef 100644 --- a/opensm/opensm/osm_lid_mgr.c +++ b/opensm/opensm/osm_lid_mgr.c @@ -47,7 +47,7 @@ * ALGORITHM: * * 0. we define a function to obtain the correct port lid: - * __osm_lid_mgr_get_port_lid( p_mgr, port, &min_lid ): + * lid_mgr_get_port_lid( p_mgr, port, &min_lid ): * 0.1 if the port info lid matches the guid2lid return 0 * 0.2 if the port info has a lid and that range is empty in * port_lid_tbl, return 0 and update the port_lid_tbl and @@ -64,12 +64,12 @@ * 2. During SM port lid assignment: * 2.1 if reassign_lids is set, make it 2^lmc * 2.2 cleanup all port_lid_tbl and re-fill it according to guid2lid - * 2.3 call __osm_lid_mgr_get_port_lid the SM port + * 2.3 call lid_mgr_get_port_lid the SM port * 2.4 set the port info * * 3. During all other ports lid assignment: * 3.1 go through all ports in the subnet - * 3.1.1 call __osm_lid_mgr_get_port_min_lid + * 3.1.1 call lid_mgr_get_port_min_lid * 3.1.2 if a change required send the port info * 3.2 if any change send the signal PENDING... * @@ -132,7 +132,7 @@ Validate the guid to lid data by making sure that under the current LMC we did not get duplicates. If we do flag them as errors and remove the entry. **********************************************************************/ -static void __osm_lid_mgr_validate_db(IN osm_lid_mgr_t * p_mgr) +static void lid_mgr_validate_db(IN osm_lid_mgr_t * p_mgr) { cl_qlist_t guids; osm_db_guid_elem_t *p_item; @@ -271,7 +271,7 @@ osm_lid_mgr_init(IN osm_lid_mgr_t * const p_mgr, IN osm_sm_t *sm) /* we need to make sure we did not get duplicates with current lmc */ - __osm_lid_mgr_validate_db(p_mgr); + lid_mgr_validate_db(p_mgr); } Exit: @@ -279,7 +279,7 @@ Exit: return (status); } -static uint16_t __osm_trim_lid(IN uint16_t lid) +static uint16_t trim_lid(IN uint16_t lid) { if ((lid > IB_LID_UCAST_END_HO) || (lid < IB_LID_UCAST_START_HO)) return 0; @@ -292,7 +292,7 @@ static uint16_t __osm_trim_lid(IN uint16_t lid) re-calculate all empty ranges. cleanup invalid port_lid_tbl entries **********************************************************************/ -static int __osm_lid_mgr_init_sweep(IN osm_lid_mgr_t * const p_mgr) +static int lid_mgr_init_sweep(IN osm_lid_mgr_t * const p_mgr) { cl_ptr_vector_t *p_discovered_vec = &p_mgr->p_subn->port_lid_tbl; uint16_t max_defined_lid; @@ -371,8 +371,8 @@ static int __osm_lid_mgr_init_sweep(IN osm_lid_mgr_t * const p_mgr) p_port != (osm_port_t *) cl_qmap_end(p_port_guid_tbl); p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item)) { osm_port_get_lid_range_ho(p_port, &disc_min_lid, &disc_max_lid); - disc_min_lid = __osm_trim_lid(disc_min_lid); - disc_max_lid = __osm_trim_lid(disc_max_lid); + disc_min_lid = trim_lid(disc_min_lid); + disc_max_lid = trim_lid(disc_max_lid); for (lid = disc_min_lid; lid <= disc_max_lid; lid++) cl_ptr_vector_set(p_discovered_vec, lid, p_port); /* make sure the guid2lid entry is valid. If not, clean it. */ @@ -581,9 +581,9 @@ AfterScanningLids: check if the given range of lids is free **********************************************************************/ static boolean_t -__osm_lid_mgr_is_range_not_persistent(IN osm_lid_mgr_t * const p_mgr, - IN const uint16_t lid, - IN const uint16_t num_lids) +lid_mgr_is_range_not_persistent(IN osm_lid_mgr_t * const p_mgr, + IN const uint16_t lid, + IN const uint16_t num_lids) { uint16_t i; const uint8_t start_lid = (uint8_t) (1 << p_mgr->p_subn->opt.lmc); @@ -602,10 +602,10 @@ __osm_lid_mgr_is_range_not_persistent(IN osm_lid_mgr_t * const p_mgr, find a free lid range **********************************************************************/ static void -__osm_lid_mgr_find_free_lid_range(IN osm_lid_mgr_t * const p_mgr, - IN const uint8_t num_lids, - OUT uint16_t * const p_min_lid, - OUT uint16_t * const p_max_lid) +lid_mgr_find_free_lid_range(IN osm_lid_mgr_t * const p_mgr, + IN const uint8_t num_lids, + OUT uint16_t * const p_min_lid, + OUT uint16_t * const p_max_lid) { uint16_t lid; cl_list_item_t *p_item; @@ -667,8 +667,8 @@ __osm_lid_mgr_find_free_lid_range(IN osm_lid_mgr_t * const p_mgr, /********************************************************************** **********************************************************************/ static void -__osm_lid_mgr_cleanup_discovered_port_lid_range(IN osm_lid_mgr_t * p_mgr, - IN osm_port_t * p_port) +lid_mgr_cleanup_discovered_port_lid_range(IN osm_lid_mgr_t * p_mgr, + IN osm_port_t * p_port) { cl_ptr_vector_t *p_discovered_vec = &p_mgr->p_subn->port_lid_tbl; uint16_t lid, min_lid, max_lid; @@ -676,8 +676,8 @@ __osm_lid_mgr_cleanup_discovered_port_lid_range(IN osm_lid_mgr_t * p_mgr, (uint16_t) (cl_ptr_vector_get_size(p_discovered_vec)); osm_port_get_lid_range_ho(p_port, &min_lid, &max_lid); - min_lid = __osm_trim_lid(min_lid); - max_lid = __osm_trim_lid(max_lid); + min_lid = trim_lid(min_lid); + max_lid = trim_lid(max_lid); for (lid = min_lid; lid <= max_lid; lid++) { if ((lid < max_tbl_lid) && (p_port == @@ -695,10 +695,10 @@ __osm_lid_mgr_cleanup_discovered_port_lid_range(IN osm_lid_mgr_t * p_mgr, port_lid_tbl and guid2lid, return 1 to flag a change required. **********************************************************************/ static int -__osm_lid_mgr_get_port_lid(IN osm_lid_mgr_t * const p_mgr, - IN osm_port_t * const p_port, - OUT uint16_t * const p_min_lid, - OUT uint16_t * const p_max_lid) +lid_mgr_get_port_lid(IN osm_lid_mgr_t * const p_mgr, + IN osm_port_t * const p_port, + OUT uint16_t * const p_min_lid, + OUT uint16_t * const p_max_lid) { uint16_t lid, min_lid, max_lid; uint64_t guid; @@ -736,8 +736,7 @@ __osm_lid_mgr_get_port_lid(IN osm_lid_mgr_t * const p_mgr, "does not match its known lid:%u\n", guid, cl_ntoh16(osm_port_get_base_lid(p_port)), min_lid); - __osm_lid_mgr_cleanup_discovered_port_lid_range(p_mgr, - p_port); + lid_mgr_cleanup_discovered_port_lid_range(p_mgr, p_port); /* we still need to send the setting to the target port */ lid_changed = 1; goto Exit; @@ -757,7 +756,7 @@ __osm_lid_mgr_get_port_lid(IN osm_lid_mgr_t * const p_mgr, /* make sure lid is valid */ if ((min_lid & lmc_mask) == min_lid) { /* is it free */ - if (__osm_lid_mgr_is_range_not_persistent + if (lid_mgr_is_range_not_persistent (p_mgr, min_lid, num_lids)) { *p_min_lid = min_lid; *p_max_lid = min_lid + num_lids - 1; @@ -780,11 +779,10 @@ __osm_lid_mgr_get_port_lid(IN osm_lid_mgr_t * const p_mgr, AssignLid: /* first cleanup the existing discovered lid range */ - __osm_lid_mgr_cleanup_discovered_port_lid_range(p_mgr, p_port); + lid_mgr_cleanup_discovered_port_lid_range(p_mgr, p_port); /* find an empty space */ - __osm_lid_mgr_find_free_lid_range(p_mgr, num_lids, p_min_lid, - p_max_lid); + lid_mgr_find_free_lid_range(p_mgr, num_lids, p_min_lid, p_max_lid); OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, "0x%016" PRIx64 " assigned a new lid range:[%u-%u]\n", guid, *p_min_lid, *p_max_lid); @@ -809,8 +807,8 @@ Exit: Set to INIT the remote port of the given physical port **********************************************************************/ static void -__osm_lid_mgr_set_remote_pi_state_to_init(IN osm_lid_mgr_t * const p_mgr, - IN osm_physp_t * const p_physp) +lid_mgr_set_remote_pi_state_to_init(IN osm_lid_mgr_t * const p_mgr, + IN osm_physp_t * const p_physp) { osm_physp_t *p_rem_physp = osm_physp_get_remote(p_physp); @@ -824,10 +822,10 @@ __osm_lid_mgr_set_remote_pi_state_to_init(IN osm_lid_mgr_t * const p_mgr, /********************************************************************** **********************************************************************/ static boolean_t -__osm_lid_mgr_set_physp_pi(IN osm_lid_mgr_t * const p_mgr, - IN osm_port_t * const p_port, - IN osm_physp_t * const p_physp, - IN ib_net16_t const lid) +lid_mgr_set_physp_pi(IN osm_lid_mgr_t * const p_mgr, + IN osm_port_t * const p_port, + IN osm_physp_t * const p_physp, + IN ib_net16_t const lid) { uint8_t payload[IB_SMP_DATA_SIZE]; ib_port_info_t *p_pi = (ib_port_info_t *) payload; @@ -1002,8 +1000,7 @@ __osm_lid_mgr_set_physp_pi(IN osm_lid_mgr_t * const p_mgr, fact that the remote port is also going through "down" state into "init"... */ - __osm_lid_mgr_set_remote_pi_state_to_init(p_mgr, - p_physp); + lid_mgr_set_remote_pi_state_to_init(p_mgr, p_physp); ib_port_info_set_port_state(p_pi, IB_LINK_DOWN); if (ib_port_info_get_port_state(p_pi) != @@ -1093,7 +1090,7 @@ Exit: Lock must already be held. **********************************************************************/ static boolean_t -__osm_lid_mgr_process_our_sm_node(IN osm_lid_mgr_t * const p_mgr) +lid_mgr_process_our_sm_node(IN osm_lid_mgr_t * const p_mgr) { osm_port_t *p_port; uint16_t min_lid_ho; @@ -1123,7 +1120,7 @@ __osm_lid_mgr_process_our_sm_node(IN osm_lid_mgr_t * const p_mgr) configure the SM with a LID that has non-zero bits, even after LMC masking by hardware. */ - __osm_lid_mgr_get_port_lid(p_mgr, p_port, &min_lid_ho, &max_lid_ho); + lid_mgr_get_port_lid(p_mgr, p_port, &min_lid_ho, &max_lid_ho); OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, "Current base LID is %u\n", min_lid_ho); /* @@ -1140,8 +1137,8 @@ __osm_lid_mgr_process_our_sm_node(IN osm_lid_mgr_t * const p_mgr) /* Set the PortInfo the Physical Port associated with this Port. */ - __osm_lid_mgr_set_physp_pi(p_mgr, p_port, p_port->p_physp, - cl_hton16(min_lid_ho)); + lid_mgr_set_physp_pi(p_mgr, p_port, p_port->p_physp, + cl_hton16(min_lid_ho)); Exit: OSM_LOG_EXIT(p_mgr->p_log); @@ -1162,13 +1159,13 @@ osm_signal_t osm_lid_mgr_process_sm(IN osm_lid_mgr_t * const p_mgr) /* initialize the port_lid_tbl and empty ranges list following the persistent db */ - __osm_lid_mgr_init_sweep(p_mgr); + lid_mgr_init_sweep(p_mgr); /* Set the send_set_reqs of the p_mgr to FALSE, and we'll see if any set requests were sent. If not - can signal OSM_SIGNAL_DONE */ p_mgr->send_set_reqs = FALSE; - if (__osm_lid_mgr_process_our_sm_node(p_mgr) == FALSE) + if (lid_mgr_process_our_sm_node(p_mgr) == FALSE) /* The initialization failed */ signal = OSM_SIGNAL_DONE; @@ -1183,7 +1180,7 @@ osm_signal_t osm_lid_mgr_process_sm(IN osm_lid_mgr_t * const p_mgr) /********************************************************************** 1 go through all ports in the subnet. - 1.1 call __osm_lid_mgr_get_port_min_lid + 1.1 call lid_mgr_get_port_min_lid 1.2 if a change is required send the port info 2 if any change send the signal PENDING... **********************************************************************/ @@ -1234,11 +1231,10 @@ osm_signal_t osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * const p_mgr) /* get the port lid range - we need to send it on first active sweep or if there was a change (the result of - __osm_lid_mgr_get_port_lid) + lid_mgr_get_port_lid) */ lid_changed = - __osm_lid_mgr_get_port_lid(p_mgr, p_port, &min_lid_ho, - &max_lid_ho); + lid_mgr_get_port_lid(p_mgr, p_port, &min_lid_ho, &max_lid_ho); /* we can call the function to update the port info as it known to look for any field change and will only send an updated @@ -1249,7 +1245,7 @@ osm_signal_t osm_lid_mgr_process_subnet(IN osm_lid_mgr_t * const p_mgr) min_lid_ho, max_lid_ho); /* the proc returns the fact it sent a set port info */ - if (__osm_lid_mgr_set_physp_pi + if (lid_mgr_set_physp_pi (p_mgr, p_port, p_port->p_physp, cl_hton16(min_lid_ho))) p_mgr->send_set_reqs = TRUE; } /* all ports */ diff --git a/opensm/opensm/osm_mcast_mgr.c b/opensm/opensm/osm_mcast_mgr.c index e42be7b..3307e26 100644 --- a/opensm/opensm/osm_mcast_mgr.c +++ b/opensm/opensm/osm_mcast_mgr.c @@ -65,8 +65,7 @@ typedef struct osm_mcast_work_obj { /********************************************************************** **********************************************************************/ -static osm_mcast_work_obj_t *__osm_mcast_work_obj_new(IN const osm_port_t * - const p_port) +static osm_mcast_work_obj_t *mcast_work_obj_new(IN const osm_port_t * const p_port) { osm_mcast_work_obj_t *p_obj; @@ -86,7 +85,7 @@ static osm_mcast_work_obj_t *__osm_mcast_work_obj_new(IN const osm_port_t * /********************************************************************** **********************************************************************/ -static void __osm_mcast_work_obj_delete(IN osm_mcast_work_obj_t * p_wobj) +static void mcast_work_obj_delete(IN osm_mcast_work_obj_t * p_wobj) { free(p_wobj); } @@ -94,14 +93,14 @@ static void __osm_mcast_work_obj_delete(IN osm_mcast_work_obj_t * p_wobj) /********************************************************************** Recursively remove nodes from the tree *********************************************************************/ -static void __osm_mcast_mgr_purge_tree_node(IN osm_mtree_node_t * p_mtn) +static void mcast_mgr_purge_tree_node(IN osm_mtree_node_t * p_mtn) { uint8_t i; for (i = 0; i < p_mtn->max_children; i++) { if (p_mtn->child_array[i] && (p_mtn->child_array[i] != OSM_MTREE_LEAF)) - __osm_mcast_mgr_purge_tree_node(p_mtn->child_array[i]); + mcast_mgr_purge_tree_node(p_mtn->child_array[i]); p_mtn->child_array[i] = NULL; @@ -113,12 +112,12 @@ static void __osm_mcast_mgr_purge_tree_node(IN osm_mtree_node_t * p_mtn) /********************************************************************** **********************************************************************/ static void -__osm_mcast_mgr_purge_tree(osm_sm_t * sm, IN osm_mgrp_t * const p_mgrp) +mcast_mgr_purge_tree(osm_sm_t * sm, IN osm_mgrp_t * const p_mgrp) { OSM_LOG_ENTER(sm->p_log); if (p_mgrp->p_root) - __osm_mcast_mgr_purge_tree_node(p_mgrp->p_root); + mcast_mgr_purge_tree_node(p_mgrp->p_root); p_mgrp->p_root = NULL; @@ -248,9 +247,8 @@ osm_mcast_mgr_compute_max_hops(osm_sm_t * sm, a switch with the lowest average hop count to the members of the multicast group. **********************************************************************/ -static osm_switch_t *__osm_mcast_mgr_find_optimal_switch(osm_sm_t * sm, - const osm_mgrp_t * - const p_mgrp) +static osm_switch_t *mcast_mgr_find_optimal_switch(osm_sm_t * sm, + const osm_mgrp_t * const p_mgrp) { cl_qmap_t *p_sw_tbl; const osm_switch_t *p_sw; @@ -306,9 +304,8 @@ static osm_switch_t *__osm_mcast_mgr_find_optimal_switch(osm_sm_t * sm, /********************************************************************** This function returns the existing or optimal root swtich for the tree. **********************************************************************/ -static osm_switch_t *__osm_mcast_mgr_find_root_switch(osm_sm_t * sm, - const osm_mgrp_t * - const p_mgrp) +static osm_switch_t *mcast_mgr_find_root_switch(osm_sm_t * sm, + const osm_mgrp_t * const p_mgrp) { const osm_switch_t *p_sw = NULL; @@ -320,7 +317,7 @@ static osm_switch_t *__osm_mcast_mgr_find_root_switch(osm_sm_t * sm, the root will be always on the first switch attached to it. - Very bad ... */ - p_sw = __osm_mcast_mgr_find_optimal_switch(sm, p_mgrp); + p_sw = mcast_mgr_find_optimal_switch(sm, p_mgrp); OSM_LOG_EXIT(sm->p_log); return ((osm_switch_t *) p_sw); @@ -329,7 +326,7 @@ static osm_switch_t *__osm_mcast_mgr_find_root_switch(osm_sm_t * sm, /********************************************************************** **********************************************************************/ static osm_signal_t -__osm_mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * const p_sw) +mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * const p_sw) { osm_node_t *p_node; osm_dr_path_t *p_path; @@ -403,12 +400,9 @@ __osm_mcast_mgr_set_tbl(osm_sm_t * sm, IN osm_switch_t * const p_sw) contains the group members that must be routed from this switch. **********************************************************************/ static void -__osm_mcast_mgr_subdivide(osm_sm_t * sm, - osm_mgrp_t * const p_mgrp, - osm_switch_t * const p_sw, - cl_qlist_t * const p_list, - cl_qlist_t * const list_array, - uint8_t const array_size) +mcast_mgr_subdivide(osm_sm_t * sm, osm_mgrp_t * const p_mgrp, + osm_switch_t * const p_sw, cl_qlist_t * const p_list, + cl_qlist_t * const list_array, uint8_t const array_size) { uint8_t port_num; uint16_t mlid_ho; @@ -453,7 +447,7 @@ __osm_mcast_mgr_subdivide(osm_sm_t * sm, cl_ntoh16(osm_port_get_base_lid (p_wobj->p_port))); - __osm_mcast_work_obj_delete(p_wobj); + mcast_work_obj_delete(p_wobj); continue; } @@ -468,7 +462,7 @@ __osm_mcast_mgr_subdivide(osm_sm_t * sm, cl_ntoh16(osm_port_get_base_lid (p_wobj->p_port))); - __osm_mcast_work_obj_delete(p_wobj); + mcast_work_obj_delete(p_wobj); /* This is means OpenSM has a bug. */ CL_ASSERT(FALSE); @@ -483,7 +477,7 @@ __osm_mcast_mgr_subdivide(osm_sm_t * sm, /********************************************************************** **********************************************************************/ -static void __osm_mcast_mgr_purge_list(osm_sm_t * sm, cl_qlist_t * const p_list) +static void mcast_mgr_purge_list(osm_sm_t * sm, cl_qlist_t * const p_list) { osm_mcast_work_obj_t *p_wobj; @@ -494,7 +488,7 @@ static void __osm_mcast_mgr_purge_list(osm_sm_t * sm, cl_qlist_t * const p_list) OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A06: " "Unable to route for port 0x%" PRIx64 "\n", osm_port_get_guid(p_wobj->p_port)); - __osm_mcast_work_obj_delete(p_wobj); + mcast_work_obj_delete(p_wobj); } OSM_LOG_EXIT(sm->p_log); @@ -507,13 +501,13 @@ static void __osm_mcast_mgr_purge_list(osm_sm_t * sm, cl_qlist_t * const p_list) The function returns the newly created mtree node element. **********************************************************************/ -static osm_mtree_node_t *__osm_mcast_mgr_branch(osm_sm_t * sm, - osm_mgrp_t * const p_mgrp, - osm_switch_t * const p_sw, - cl_qlist_t * const p_list, - uint8_t depth, - uint8_t const upstream_port, - uint8_t * const p_max_depth) +static osm_mtree_node_t *mcast_mgr_branch(osm_sm_t * sm, + osm_mgrp_t * const p_mgrp, + osm_switch_t * const p_sw, + cl_qlist_t * const p_list, + uint8_t depth, + uint8_t const upstream_port, + uint8_t * const p_max_depth) { uint8_t max_children; osm_mtree_node_t *p_mtn = NULL; @@ -550,7 +544,7 @@ static osm_mtree_node_t *__osm_mcast_mgr_branch(osm_sm_t * sm, OSM_LOG(sm->p_log, OSM_LOG_ERROR, "Maximal hops number is reached for MLID 0x%x." " Break processing.", mlid_ho); - __osm_mcast_mgr_purge_list(sm, p_list); + mcast_mgr_purge_list(sm, p_list); goto Exit; } @@ -570,7 +564,7 @@ static osm_mtree_node_t *__osm_mcast_mgr_branch(osm_sm_t * sm, /* Deallocate all the work objects on this branch of the tree. */ - __osm_mcast_mgr_purge_list(sm, p_list); + mcast_mgr_purge_list(sm, p_list); goto Exit; } @@ -586,7 +580,7 @@ static osm_mtree_node_t *__osm_mcast_mgr_branch(osm_sm_t * sm, /* Deallocate all the work objects on this branch of the tree. */ - __osm_mcast_mgr_purge_list(sm, p_list); + mcast_mgr_purge_list(sm, p_list); goto Exit; } @@ -603,7 +597,7 @@ static osm_mtree_node_t *__osm_mcast_mgr_branch(osm_sm_t * sm, if (list_array == NULL) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A16: " "Unable to allocate list array\n"); - __osm_mcast_mgr_purge_list(sm, p_list); + mcast_mgr_purge_list(sm, p_list); goto Exit; } @@ -612,8 +606,7 @@ static osm_mtree_node_t *__osm_mcast_mgr_branch(osm_sm_t * sm, for (i = 0; i < max_children; i++) cl_qlist_init(&list_array[i]); - __osm_mcast_mgr_subdivide(sm, p_mgrp, p_sw, p_list, list_array, - max_children); + mcast_mgr_subdivide(sm, p_mgrp, p_sw, p_list, list_array, max_children); p_tbl = osm_switch_get_mcast_tbl_ptr(p_sw); @@ -674,7 +667,7 @@ static osm_mtree_node_t *__osm_mcast_mgr_branch(osm_sm_t * sm, p_wobj = (osm_mcast_work_obj_t *) cl_qlist_remove_head(p_port_list); - __osm_mcast_work_obj_delete(p_wobj); + mcast_work_obj_delete(p_wobj); continue; } @@ -696,12 +689,10 @@ static osm_mtree_node_t *__osm_mcast_mgr_branch(osm_sm_t * sm, CL_ASSERT(p_remote_physp); p_mtn->child_array[i] = - __osm_mcast_mgr_branch(sm, p_mgrp, - p_remote_node->sw, - p_port_list, depth, - osm_physp_get_port_num - (p_remote_physp), - p_max_depth); + mcast_mgr_branch(sm, p_mgrp, p_remote_node->sw, + p_port_list, depth, + osm_physp_get_port_num(p_remote_physp), + p_max_depth); } else { /* The neighbor node is not a switch, so this @@ -721,7 +712,7 @@ static osm_mtree_node_t *__osm_mcast_mgr_branch(osm_sm_t * sm, cl_ntoh64(osm_port_get_guid(p_wobj->p_port)), i); - __osm_mcast_work_obj_delete(p_wobj); + mcast_work_obj_delete(p_wobj); } } @@ -734,7 +725,7 @@ Exit: /********************************************************************** **********************************************************************/ static ib_api_status_t -__osm_mcast_mgr_build_spanning_tree(osm_sm_t * sm, osm_mgrp_t * const p_mgrp) +mcast_mgr_build_spanning_tree(osm_sm_t * sm, osm_mgrp_t * const p_mgrp) { const cl_qmap_t *p_mcm_tbl; const osm_port_t *p_port; @@ -757,7 +748,7 @@ __osm_mcast_mgr_build_spanning_tree(osm_sm_t * sm, osm_mgrp_t * const p_mgrp) on multicast forwarding table information if the user wants to preserve existing multicast routes. */ - __osm_mcast_mgr_purge_tree(sm, p_mgrp); + mcast_mgr_purge_tree(sm, p_mgrp); p_mcm_tbl = &p_mgrp->mcm_port_tbl; num_ports = cl_qmap_count(p_mcm_tbl); @@ -784,7 +775,7 @@ __osm_mcast_mgr_build_spanning_tree(osm_sm_t * sm, osm_mgrp_t * const p_mgrp) Locate the switch around which to create the spanning tree for this multicast group. */ - p_sw = __osm_mcast_mgr_find_root_switch(sm, p_mgrp); + p_sw = mcast_mgr_find_root_switch(sm, p_mgrp); if (p_sw == NULL) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A08: " "Unable to locate a suitable switch for group 0x%X\n", @@ -815,7 +806,7 @@ __osm_mcast_mgr_build_spanning_tree(osm_sm_t * sm, osm_mgrp_t * const p_mgrp) continue; } - p_wobj = __osm_mcast_work_obj_new(p_port); + p_wobj = mcast_work_obj_new(p_port); if (p_wobj == NULL) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A10: " "Insufficient memory to route port 0x%016" @@ -828,7 +819,7 @@ __osm_mcast_mgr_build_spanning_tree(osm_sm_t * sm, osm_mgrp_t * const p_mgrp) } count = cl_qlist_count(&port_list); - p_mgrp->p_root = __osm_mcast_mgr_branch(sm, p_mgrp, p_sw, + p_mgrp->p_root = mcast_mgr_branch(sm, p_mgrp, p_sw, &port_list, 0, 0, &max_depth); OSM_LOG(sm->p_log, OSM_LOG_VERBOSE, @@ -892,7 +883,7 @@ osm_mcast_mgr_set_table(osm_sm_t * sm, /********************************************************************** **********************************************************************/ -static void __osm_mcast_mgr_clear(osm_sm_t * sm, uint16_t mlid) +static void mcast_mgr_clear(osm_sm_t * sm, uint16_t mlid) { osm_switch_t *p_sw; cl_qmap_t *p_sw_tbl; @@ -1069,12 +1060,12 @@ osm_mcast_mgr_process_tree(osm_sm_t * sm, IN osm_mgrp_t * const p_mgrp) the spanning tree which sets the mcast table bits for each port in the group. */ - __osm_mcast_mgr_clear(sm, cl_ntoh16(mlid)); + mcast_mgr_clear(sm, cl_ntoh16(mlid)); if (!p_mgrp->full_members) goto Exit; - status = __osm_mcast_mgr_build_spanning_tree(sm, p_mgrp); + status = mcast_mgr_build_spanning_tree(sm, p_mgrp); if (status != IB_SUCCESS) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 0A17: " "Unable to create spanning tree (%s)\n", @@ -1161,7 +1152,7 @@ osm_signal_t osm_mcast_mgr_process(osm_sm_t * sm) */ p_sw = (osm_switch_t *) cl_qmap_head(p_sw_tbl); while (p_sw != (osm_switch_t *) cl_qmap_end(p_sw_tbl)) { - signal = __osm_mcast_mgr_set_tbl(sm, p_sw); + signal = mcast_mgr_set_tbl(sm, p_sw); if (signal == OSM_SIGNAL_DONE_PENDING) pending_transactions = TRUE; p_sw = (osm_switch_t *) cl_qmap_next(&p_sw->map_item); @@ -1238,7 +1229,7 @@ osm_signal_t osm_mcast_mgr_process_mgroups(osm_sm_t * sm) p_sw_tbl = &sm->p_subn->sw_guid_tbl; p_sw = (osm_switch_t *) cl_qmap_head(p_sw_tbl); while (p_sw != (osm_switch_t *) cl_qmap_end(p_sw_tbl)) { - ret = __osm_mcast_mgr_set_tbl(sm, p_sw); + ret = mcast_mgr_set_tbl(sm, p_sw); if (ret == OSM_SIGNAL_DONE_PENDING) signal = ret; p_sw = (osm_switch_t *) cl_qmap_next(&p_sw->map_item); diff --git a/opensm/opensm/osm_multicast.c b/opensm/opensm/osm_multicast.c index 350fd22..5c1a0d7 100644 --- a/opensm/opensm/osm_multicast.c +++ b/opensm/opensm/osm_multicast.c @@ -264,9 +264,9 @@ osm_mgrp_is_port_present(IN const osm_mgrp_t * const p_mgrp, /********************************************************************** **********************************************************************/ static void -__osm_mgrp_apply_func_sub(const osm_mgrp_t * const p_mgrp, - const osm_mtree_node_t * const p_mtn, - osm_mgrp_func_t p_func, void *context) +mgrp_apply_func_sub(const osm_mgrp_t * const p_mgrp, + const osm_mtree_node_t * const p_mtn, + osm_mgrp_func_t p_func, void *context) { uint8_t i = 0; uint8_t max_children; @@ -279,8 +279,8 @@ __osm_mgrp_apply_func_sub(const osm_mgrp_t * const p_mgrp, for (i = 0; i < max_children; i++) { p_child_mtn = osm_mtree_node_get_child(p_mtn, i); if (p_child_mtn) - __osm_mgrp_apply_func_sub(p_mgrp, p_child_mtn, p_func, - context); + mgrp_apply_func_sub(p_mgrp, p_child_mtn, p_func, + context); } } @@ -298,5 +298,5 @@ osm_mgrp_apply_func(const osm_mgrp_t * const p_mgrp, p_mtn = p_mgrp->p_root; if (p_mtn) - __osm_mgrp_apply_func_sub(p_mgrp, p_mtn, p_func, context); + mgrp_apply_func_sub(p_mgrp, p_mtn, p_func, context); } diff --git a/opensm/opensm/osm_pkey.c b/opensm/opensm/osm_pkey.c index 4666186..1128172 100644 --- a/opensm/opensm/osm_pkey.c +++ b/opensm/opensm/osm_pkey.c @@ -287,7 +287,7 @@ osm_pkey_tbl_get_block_and_idx(IN osm_pkey_tbl_t * p_pkey_tbl, /********************************************************************** **********************************************************************/ static boolean_t -__osm_match_pkey(IN const ib_net16_t * pkey1, IN const ib_net16_t * pkey2) +match_pkey(IN const ib_net16_t * pkey1, IN const ib_net16_t * pkey2) { /* if both pkeys are not full member - this is not a match */ @@ -315,7 +315,7 @@ osm_physp_share_this_pkey(IN const osm_physp_t * const p_physp1, ib_pkey_get_base(pkey)); pkey2 = cl_map_get(&(osm_physp_get_pkey_tbl(p_physp2))->keys, ib_pkey_get_base(pkey)); - return (pkey1 && pkey2 && __osm_match_pkey(pkey1, pkey2)); + return (pkey1 && pkey2 && match_pkey(pkey1, pkey2)); } /********************************************************************** @@ -341,7 +341,7 @@ osm_physp_find_common_pkey(IN const osm_physp_t * const p_physp1, pkey1 = (ib_net16_t *) cl_map_obj(map_iter1); pkey2 = (ib_net16_t *) cl_map_obj(map_iter2); - if (__osm_match_pkey(pkey1, pkey2)) + if (match_pkey(pkey1, pkey2)) return *pkey1; /* advance the lower value if they are not equal */ diff --git a/opensm/opensm/osm_port.c b/opensm/opensm/osm_port.c index ade85c0..34b5688 100644 --- a/opensm/opensm/osm_port.c +++ b/opensm/opensm/osm_port.c @@ -249,8 +249,8 @@ osm_port_add_mgrp(IN osm_port_t * const p_port, IN const ib_net16_t mlid) /********************************************************************** **********************************************************************/ static cl_status_t -__osm_port_mgrp_find_func(IN const cl_list_item_t * const p_list_item, - IN void *context) +port_mgrp_find_func(IN const cl_list_item_t * const p_list_item, + IN void *context) { if (*((ib_net16_t *) context) == ((osm_mcm_info_t *) p_list_item)->mlid) return (CL_SUCCESS); @@ -265,8 +265,8 @@ osm_port_remove_mgrp(IN osm_port_t * const p_port, IN const ib_net16_t mlid) { cl_list_item_t *p_mcm; - p_mcm = cl_qlist_find_from_head(&p_port->mcm_list, - __osm_port_mgrp_find_func, &mlid); + p_mcm = cl_qlist_find_from_head(&p_port->mcm_list, port_mgrp_find_func, + &mlid); if (p_mcm != cl_qlist_end(&p_port->mcm_list)) { cl_qlist_remove_item(&p_port->mcm_list, p_mcm); @@ -399,7 +399,7 @@ osm_physp_calc_link_op_vls(IN osm_log_t * p_log, return (op_vls); } -static inline uint64_t __osm_ptr_to_key(void const *p) +static inline uint64_t ptr_to_key(void const *p) { uint64_t k = 0; @@ -407,7 +407,7 @@ static inline uint64_t __osm_ptr_to_key(void const *p) return k; } -static inline void *__osm_key_to_ptr(uint64_t k) +static inline void *key_to_ptr(uint64_t k) { void *p = 0; @@ -421,10 +421,9 @@ static inline void *__osm_key_to_ptr(uint64_t k) last phys ports (going into the first switch and into the target port). **********************************************************************/ static cl_status_t -__osm_physp_get_dr_physp_set(IN osm_log_t * p_log, - IN osm_subn_t const *p_subn, - IN osm_dr_path_t const *p_path, - OUT cl_map_t * p_physp_map) +physp_get_dr_physp_set(IN osm_log_t * p_log, IN osm_subn_t const *p_subn, + IN osm_dr_path_t const *p_path, + OUT cl_map_t * p_physp_map) { osm_port_t *p_port; osm_physp_t *p_physp; @@ -466,8 +465,7 @@ __osm_physp_get_dr_physp_set(IN osm_log_t * p_log, /* we track the ports we go out along the path */ if (hop > 1) - cl_map_insert(p_physp_map, __osm_ptr_to_key(p_physp), - NULL); + cl_map_insert(p_physp_map, ptr_to_key(p_physp), NULL); OSM_LOG(p_log, OSM_LOG_DEBUG, "Traversed through node: 0x%016" PRIx64 @@ -494,9 +492,9 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_physp_update_new_dr_path(IN osm_physp_t const *p_dest_physp, - IN cl_map_t * p_visited_map, - IN osm_bind_handle_t * h_bind) +physp_update_new_dr_path(IN osm_physp_t const *p_dest_physp, + IN cl_map_t * p_visited_map, + IN osm_bind_handle_t * h_bind) { cl_list_t tmpPortsList; osm_physp_t *p_physp, *p_src_physp = NULL; @@ -510,20 +508,20 @@ __osm_physp_update_new_dr_path(IN osm_physp_t const *p_dest_physp, cl_list_insert_head(&tmpPortsList, p_dest_physp); /* get the output port where we need to come from */ p_physp = (osm_physp_t *) cl_map_get(p_visited_map, - __osm_ptr_to_key(p_dest_physp)); + ptr_to_key(p_dest_physp)); while (p_physp != NULL) { cl_list_insert_head(&tmpPortsList, p_physp); /* get the input port through where we reached the output port */ p_src_physp = p_physp; p_physp = (osm_physp_t *) cl_map_get(p_visited_map, - __osm_ptr_to_key(p_physp)); + ptr_to_key(p_physp)); /* if we reached a null p_physp - this means we are at the begining of the path. Break. */ if (p_physp == NULL) break; /* get the output port */ p_physp = (osm_physp_t *) cl_map_get(p_visited_map, - __osm_ptr_to_key(p_physp)); + ptr_to_key(p_physp)); } memset(path_array, 0, sizeof(path_array)); @@ -574,7 +572,7 @@ osm_physp_replace_dr_path_with_alternate_dr_path(IN osm_log_t * p_log, cl_map_construct(&visited_map); cl_map_init(&visited_map, 4); p_dr_path = osm_physp_get_dr_path_ptr(p_dest_physp); - __osm_physp_get_dr_physp_set(p_log, p_subn, p_dr_path, &physp_map); + physp_get_dr_physp_set(p_log, p_subn, p_dr_path, &physp_map); /* BFS from OSM port until we find the target physp but avoid @@ -643,21 +641,20 @@ osm_physp_replace_dr_path_with_alternate_dr_path(IN osm_log_t * p_log, if (p_remote_physp && p_remote_physp != p_physp && cl_map_get(&physp_map, - __osm_ptr_to_key(p_remote_physp)) + ptr_to_key(p_remote_physp)) == NULL && cl_map_get(&visited_map, - __osm_ptr_to_key - (p_remote_physp)) == NULL) { + ptr_to_key(p_remote_physp)) + == NULL) { /* Insert the port into the visited_map, and save its source port */ cl_map_insert(&visited_map, - __osm_ptr_to_key - (p_remote_physp), + ptr_to_key(p_remote_physp), p_physp); /* Is this the p_dest_physp? */ if (p_remote_physp == p_dest_physp) { /* update the new dr path */ - __osm_physp_update_new_dr_path + physp_update_new_dr_path (p_dest_physp, &visited_map, h_bind); reached_dest = TRUE; diff --git a/opensm/opensm/osm_sa_class_port_info.c b/opensm/opensm/osm_sa_class_port_info.c index a3a2782..776e698 100644 --- a/opensm/opensm/osm_sa_class_port_info.c +++ b/opensm/opensm/osm_sa_class_port_info.c @@ -69,8 +69,7 @@ const static uint32_t __msecs_to_rtv_table[MAX_MSECS_TO_RTV] = { /********************************************************************** **********************************************************************/ static void -__osm_cpi_rcv_respond(IN osm_sa_t * sa, - IN const osm_madw_t * const p_madw) +cpi_rcv_respond(IN osm_sa_t * sa, IN const osm_madw_t * const p_madw) { osm_madw_t *p_resp_madw; const ib_sa_mad_t *p_sa_mad; @@ -202,11 +201,9 @@ void osm_cpi_rcv_process(IN void *context, IN void *data) CL_ASSERT(p_sa_mad->attr_id == IB_MAD_ATTR_CLASS_PORT_INFO); - /* - CLASS PORT INFO does not really look on the SMDB - no lock required. - */ + /* CLASS PORT INFO does not really look at the SMDB - no lock required. */ - __osm_cpi_rcv_respond(sa, p_madw); + cpi_rcv_respond(sa, p_madw); Exit: OSM_LOG_EXIT(sa->p_log); diff --git a/opensm/opensm/osm_sa_guidinfo_record.c b/opensm/opensm/osm_sa_guidinfo_record.c index 39dae72..df98fa1 100644 --- a/opensm/opensm/osm_sa_guidinfo_record.c +++ b/opensm/opensm/osm_sa_guidinfo_record.c @@ -73,13 +73,12 @@ typedef struct osm_gir_search_ctxt { /********************************************************************** **********************************************************************/ static ib_api_status_t -__osm_gir_rcv_new_gir(IN osm_sa_t * sa, - IN const osm_node_t * const p_node, - IN cl_qlist_t * const p_list, - IN ib_net64_t const match_port_guid, - IN ib_net16_t const match_lid, - IN const osm_physp_t * const p_req_physp, - IN uint8_t const block_num) +gir_rcv_new_gir(IN osm_sa_t * sa, IN const osm_node_t * const p_node, + IN cl_qlist_t * const p_list, + IN ib_net64_t const match_port_guid, + IN ib_net16_t const match_lid, + IN const osm_physp_t * const p_req_physp, + IN uint8_t const block_num) { osm_gir_item_t *p_rec_item; ib_api_status_t status = IB_SUCCESS; @@ -116,13 +115,12 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sa_gir_create_gir(IN osm_sa_t * sa, - IN osm_node_t * const p_node, - IN cl_qlist_t * const p_list, - IN ib_net64_t const match_port_guid, - IN ib_net16_t const match_lid, - IN const osm_physp_t * const p_req_physp, - IN uint8_t const match_block_num) +sa_gir_create_gir(IN osm_sa_t * sa, IN osm_node_t * const p_node, + IN cl_qlist_t * const p_list, + IN ib_net64_t const match_port_guid, + IN ib_net16_t const match_lid, + IN const osm_physp_t * const p_req_physp, + IN uint8_t const match_block_num) { const osm_physp_t *p_physp; uint8_t port_num; @@ -206,9 +204,9 @@ __osm_sa_gir_create_gir(IN osm_sa_t * sa, for (block_num = start_block_num; block_num <= end_block_num; block_num++) - __osm_gir_rcv_new_gir(sa, p_node, p_list, port_guid, - cl_ntoh16(base_lid_ho), p_physp, - block_num); + gir_rcv_new_gir(sa, p_node, p_list, port_guid, + cl_ntoh16(base_lid_ho), p_physp, + block_num); } @@ -218,8 +216,7 @@ __osm_sa_gir_create_gir(IN osm_sa_t * sa, /********************************************************************** **********************************************************************/ static void -__osm_sa_gir_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, - IN void *context) +sa_gir_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, IN void *context) { const osm_gir_search_ctxt_t *const p_ctxt = (osm_gir_search_ctxt_t *) context; @@ -285,9 +282,8 @@ __osm_sa_gir_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, goto Exit; } - __osm_sa_gir_create_gir(sa, p_node, p_ctxt->p_list, - match_port_guid, match_lid, p_req_physp, - match_block_num); + sa_gir_create_gir(sa, p_node, p_ctxt->p_list, match_port_guid, + match_lid, p_req_physp, match_block_num); Exit: OSM_LOG_EXIT(p_ctxt->sa->p_log); @@ -350,8 +346,8 @@ void osm_gir_rcv_process(IN void *ctx, IN void *data) cl_plock_acquire(sa->p_lock); - cl_qmap_apply_func(&sa->p_subn->node_guid_tbl, - __osm_sa_gir_by_comp_mask_cb, &context); + cl_qmap_apply_func(&sa->p_subn->node_guid_tbl, sa_gir_by_comp_mask_cb, + &context); cl_plock_release(sa->p_lock); diff --git a/opensm/opensm/osm_sa_informinfo.c b/opensm/opensm/osm_sa_informinfo.c index fb81008..c793c3a 100644 --- a/opensm/opensm/osm_sa_informinfo.c +++ b/opensm/opensm/osm_sa_informinfo.c @@ -210,8 +210,7 @@ with an InformInfo attribute that is a copy of the data in the Set(InformInfo) request. **********************************************************************/ static void -__osm_infr_rcv_respond(IN osm_sa_t * sa, - IN osm_madw_t * const p_madw) +infr_rcv_respond(IN osm_sa_t * sa, IN osm_madw_t * const p_madw) { cl_qlist_t rec_list; osm_iir_item_t *item; @@ -244,9 +243,9 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sa_inform_info_rec_by_comp_mask(IN osm_sa_t * sa, - IN const osm_infr_t * const p_infr, - osm_iir_search_ctxt_t * const p_ctxt) +sa_inform_info_rec_by_comp_mask(IN osm_sa_t * sa, + IN const osm_infr_t * const p_infr, + osm_iir_search_ctxt_t * const p_ctxt) { const ib_inform_info_record_t *p_rcvd_rec = NULL; ib_net64_t comp_mask; @@ -313,13 +312,13 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sa_inform_info_rec_by_comp_mask_cb(IN cl_list_item_t * const p_list_item, - IN void *context) +sa_inform_info_rec_by_comp_mask_cb(IN cl_list_item_t * const p_list_item, + IN void *context) { const osm_infr_t *const p_infr = (osm_infr_t *) p_list_item; osm_iir_search_ctxt_t *const p_ctxt = (osm_iir_search_ctxt_t *) context; - __osm_sa_inform_info_rec_by_comp_mask(p_ctxt->sa, p_infr, p_ctxt); + sa_inform_info_rec_by_comp_mask(p_ctxt->sa, p_infr, p_ctxt); } /********************************************************************** @@ -379,7 +378,7 @@ osm_infr_rcv_process_get_method(IN osm_sa_t * sa, cl_plock_acquire(sa->p_lock); cl_qlist_apply_func(&sa->p_subn->sa_infr_list, - __osm_sa_inform_info_rec_by_comp_mask_cb, &context); + sa_inform_info_rec_by_comp_mask_cb, &context); cl_plock_release(sa->p_lock); @@ -555,7 +554,7 @@ osm_infr_rcv_process_set_method(IN osm_sa_t * sa, cl_plock_release(sa->p_lock); /* send the success response */ - __osm_infr_rcv_respond(sa, p_madw); + infr_rcv_respond(sa, p_madw); Exit: OSM_LOG_EXIT(sa->p_log); diff --git a/opensm/opensm/osm_sa_lft_record.c b/opensm/opensm/osm_sa_lft_record.c index d84a6a5..ef90028 100644 --- a/opensm/opensm/osm_sa_lft_record.c +++ b/opensm/opensm/osm_sa_lft_record.c @@ -70,10 +70,9 @@ typedef struct osm_lftr_search_ctxt { /********************************************************************** **********************************************************************/ static ib_api_status_t -__osm_lftr_rcv_new_lftr(IN osm_sa_t * sa, - IN const osm_switch_t * const p_sw, - IN cl_qlist_t * const p_list, - IN ib_net16_t const lid, IN uint16_t const block) +lftr_rcv_new_lftr(IN osm_sa_t * sa, IN const osm_switch_t * const p_sw, + IN cl_qlist_t * const p_list, + IN ib_net16_t const lid, IN uint16_t const block) { osm_lftr_item_t *p_rec_item; ib_api_status_t status = IB_SUCCESS; @@ -112,8 +111,7 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_lftr_rcv_by_comp_mask(IN cl_map_item_t * const p_map_item, - IN void *context) +lftr_rcv_by_comp_mask(IN cl_map_item_t * const p_map_item, IN void *context) { const osm_lftr_search_ctxt_t *const p_ctxt = (osm_lftr_search_ctxt_t *) context; @@ -176,8 +174,8 @@ __osm_lftr_rcv_by_comp_mask(IN cl_map_item_t * const p_map_item, /* so we can add these blocks one by one ... */ for (block = min_block; block <= max_block; block++) - __osm_lftr_rcv_new_lftr(sa, p_sw, p_ctxt->p_list, - osm_port_get_base_lid(p_port), block); + lftr_rcv_new_lftr(sa, p_sw, p_ctxt->p_list, + osm_port_get_base_lid(p_port), block); } /********************************************************************** @@ -234,8 +232,8 @@ void osm_lftr_rcv_process(IN void *ctx, IN void *data) cl_plock_acquire(sa->p_lock); /* Go over all switches */ - cl_qmap_apply_func(&sa->p_subn->sw_guid_tbl, - __osm_lftr_rcv_by_comp_mask, &context); + cl_qmap_apply_func(&sa->p_subn->sw_guid_tbl, lftr_rcv_by_comp_mask, + &context); cl_plock_release(sa->p_lock); diff --git a/opensm/opensm/osm_sa_link_record.c b/opensm/opensm/osm_sa_link_record.c index 26d62e0..4df4794 100644 --- a/opensm/opensm/osm_sa_link_record.c +++ b/opensm/opensm/osm_sa_link_record.c @@ -63,11 +63,9 @@ typedef struct osm_lr_item { /********************************************************************** **********************************************************************/ static void -__osm_lr_rcv_build_physp_link(IN osm_sa_t * sa, - IN const ib_net16_t from_lid, - IN const ib_net16_t to_lid, - IN const uint8_t from_port, - IN const uint8_t to_port, IN cl_qlist_t * p_list) +lr_rcv_build_physp_link(IN osm_sa_t * sa, IN const ib_net16_t from_lid, + IN const ib_net16_t to_lid, IN const uint8_t from_port, + IN const uint8_t to_port, IN cl_qlist_t * p_list) { osm_lr_item_t *p_lr_item; @@ -105,13 +103,13 @@ static ib_net16_t get_base_lid(IN const osm_physp_t * p_physp) /********************************************************************** **********************************************************************/ static void -__osm_lr_rcv_get_physp_link(IN osm_sa_t * sa, - IN const ib_link_record_t * const p_lr, - IN const osm_physp_t * p_src_physp, - IN const osm_physp_t * p_dest_physp, - IN const ib_net64_t comp_mask, - IN cl_qlist_t * const p_list, - IN const osm_physp_t * p_req_physp) +lr_rcv_get_physp_link(IN osm_sa_t * sa, + IN const ib_link_record_t * const p_lr, + IN const osm_physp_t * p_src_physp, + IN const osm_physp_t * p_dest_physp, + IN const ib_net64_t comp_mask, + IN cl_qlist_t * const p_list, + IN const osm_physp_t * p_req_physp) { uint8_t src_port_num; uint8_t dest_port_num; @@ -198,8 +196,8 @@ __osm_lr_rcv_get_physp_link(IN osm_sa_t * sa, dest_port_num); - __osm_lr_rcv_build_physp_link(sa, from_base_lid, to_base_lid, - src_port_num, dest_port_num, p_list); + lr_rcv_build_physp_link(sa, from_base_lid, to_base_lid, src_port_num, + dest_port_num, p_list); Exit: OSM_LOG_EXIT(sa->p_log); @@ -208,13 +206,13 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_lr_rcv_get_port_links(IN osm_sa_t * sa, - IN const ib_link_record_t * const p_lr, - IN const osm_port_t * p_src_port, - IN const osm_port_t * p_dest_port, - IN const ib_net64_t comp_mask, - IN cl_qlist_t * const p_list, - IN const osm_physp_t * p_req_physp) +lr_rcv_get_port_links(IN osm_sa_t * sa, + IN const ib_link_record_t * const p_lr, + IN const osm_port_t * p_src_port, + IN const osm_port_t * p_dest_port, + IN const ib_net64_t comp_mask, + IN cl_qlist_t * const p_list, + IN const osm_physp_t * p_req_physp) { const osm_physp_t *p_src_physp; const osm_physp_t *p_dest_physp; @@ -251,7 +249,7 @@ __osm_lr_rcv_get_port_links(IN osm_sa_t * sa, dest_port_num); /* both physical ports should be with data */ if (p_src_physp && p_dest_physp) - __osm_lr_rcv_get_physp_link + lr_rcv_get_physp_link (sa, p_lr, p_src_physp, p_dest_physp, comp_mask, p_list, p_req_physp); @@ -272,7 +270,7 @@ __osm_lr_rcv_get_port_links(IN osm_sa_t * sa, p_node, port_num); if (p_src_physp) - __osm_lr_rcv_get_physp_link + lr_rcv_get_physp_link (sa, p_lr, p_src_physp, NULL, comp_mask, p_list, p_req_physp); @@ -287,7 +285,7 @@ __osm_lr_rcv_get_port_links(IN osm_sa_t * sa, p_node, port_num); if (p_src_physp) - __osm_lr_rcv_get_physp_link + lr_rcv_get_physp_link (sa, p_lr, p_src_physp, NULL, comp_mask, p_list, p_req_physp); @@ -310,7 +308,7 @@ __osm_lr_rcv_get_port_links(IN osm_sa_t * sa, p_node, port_num); if (p_dest_physp) - __osm_lr_rcv_get_physp_link + lr_rcv_get_physp_link (sa, p_lr, NULL, p_dest_physp, comp_mask, p_list, p_req_physp); @@ -325,7 +323,7 @@ __osm_lr_rcv_get_port_links(IN osm_sa_t * sa, p_node, port_num); if (p_dest_physp) - __osm_lr_rcv_get_physp_link + lr_rcv_get_physp_link (sa, p_lr, NULL, p_dest_physp, comp_mask, p_list, p_req_physp); @@ -346,7 +344,7 @@ __osm_lr_rcv_get_port_links(IN osm_sa_t * sa, osm_node_get_physp_ptr(p_node, port_num); if (p_src_physp) - __osm_lr_rcv_get_physp_link + lr_rcv_get_physp_link (sa, p_lr, p_src_physp, NULL, comp_mask, p_list, p_req_physp); @@ -364,10 +362,9 @@ __osm_lr_rcv_get_port_links(IN osm_sa_t * sa, Returns the SA status to return to the client. **********************************************************************/ static ib_net16_t -__osm_lr_rcv_get_end_points(IN osm_sa_t * sa, - IN const osm_madw_t * const p_madw, - OUT const osm_port_t ** const pp_src_port, - OUT const osm_port_t ** const pp_dest_port) +lr_rcv_get_end_points(IN osm_sa_t * sa, IN const osm_madw_t * const p_madw, + OUT const osm_port_t ** const pp_src_port, + OUT const osm_port_t ** const pp_dest_port) { const ib_link_record_t *p_lr; const ib_sa_mad_t *p_sa_mad; @@ -485,13 +482,11 @@ void osm_lr_rcv_process(IN void *context, IN void *data) */ cl_plock_acquire(sa->p_lock); - sa_status = __osm_lr_rcv_get_end_points(sa, p_madw, - &p_src_port, &p_dest_port); + sa_status = lr_rcv_get_end_points(sa, p_madw, &p_src_port, &p_dest_port); if (sa_status == IB_SA_MAD_STATUS_SUCCESS) - __osm_lr_rcv_get_port_links(sa, p_lr, p_src_port, - p_dest_port, p_sa_mad->comp_mask, - &lr_list, p_req_physp); + lr_rcv_get_port_links(sa, p_lr, p_src_port, p_dest_port, + p_sa_mad->comp_mask, &lr_list, p_req_physp); cl_plock_release(sa->p_lock); diff --git a/opensm/opensm/osm_sa_mad_ctrl.c b/opensm/opensm/osm_sa_mad_ctrl.c index 49309f9..3ed763d 100644 --- a/opensm/opensm/osm_sa_mad_ctrl.c +++ b/opensm/opensm/osm_sa_mad_ctrl.c @@ -52,9 +52,9 @@ #include #include -/****f* opensm: SA/__osm_sa_mad_ctrl_disp_done_callback +/****f* opensm: SA/sa_mad_ctrl_disp_done_callback * NAME - * __osm_sa_mad_ctrl_disp_done_callback + * sa_mad_ctrl_disp_done_callback * * DESCRIPTION * This function is the Dispatcher callback that indicates @@ -63,7 +63,7 @@ * SYNOPSIS */ static void -__osm_sa_mad_ctrl_disp_done_callback(IN void *context, IN void *p_data) +sa_mad_ctrl_disp_done_callback(IN void *context, IN void *p_data) { osm_sa_mad_ctrl_t *const p_ctrl = (osm_sa_mad_ctrl_t *) context; osm_madw_t *const p_madw = (osm_madw_t *) p_data; @@ -80,9 +80,9 @@ __osm_sa_mad_ctrl_disp_done_callback(IN void *context, IN void *p_data) /************/ -/****f* opensm: SA/__osm_sa_mad_ctrl_process +/****f* opensm: SA/sa_mad_ctrl_process * NAME - * __osm_sa_mad_ctrl_process + * sa_mad_ctrl_process * * DESCRIPTION * This function handles known methods for received MADs. @@ -90,8 +90,7 @@ __osm_sa_mad_ctrl_disp_done_callback(IN void *context, IN void *p_data) * SYNOPSIS */ static void -__osm_sa_mad_ctrl_process(IN osm_sa_mad_ctrl_t * const p_ctrl, - IN osm_madw_t * p_madw) +sa_mad_ctrl_process(IN osm_sa_mad_ctrl_t * const p_ctrl, IN osm_madw_t * p_madw) { ib_sa_mad_t *p_sa_mad; cl_status_t status; @@ -232,11 +231,8 @@ __osm_sa_mad_ctrl_process(IN osm_sa_mad_ctrl_t * const p_ctrl, "Posting Dispatcher message %s\n", osm_get_disp_msg_str(msg_id)); - status = cl_disp_post(p_ctrl->h_disp, - msg_id, - p_madw, - __osm_sa_mad_ctrl_disp_done_callback, - p_ctrl); + status = cl_disp_post(p_ctrl->h_disp, msg_id, p_madw, + sa_mad_ctrl_disp_done_callback, p_ctrl); if (status != CL_SUCCESS) { OSM_LOG(p_ctrl->p_log, OSM_LOG_ERROR, "ERR 1A02: " @@ -270,9 +266,9 @@ Exit: * SEE ALSO *********/ -/****f* opensm: SA/__osm_sa_mad_ctrl_rcv_callback +/****f* opensm: SA/sa_mad_ctrl_rcv_callback * NAME - * __osm_sa_mad_ctrl_rcv_callback + * sa_mad_ctrl_rcv_callback * * DESCRIPTION * This is the callback from the transport layer for received MADs. @@ -280,9 +276,8 @@ Exit: * SYNOPSIS */ static void -__osm_sa_mad_ctrl_rcv_callback(IN osm_madw_t * p_madw, - IN void *bind_context, - IN osm_madw_t * p_req_madw) +sa_mad_ctrl_rcv_callback(IN osm_madw_t * p_madw, IN void *bind_context, + IN osm_madw_t * p_req_madw) { osm_sa_mad_ctrl_t *p_ctrl = (osm_sa_mad_ctrl_t *) bind_context; ib_sa_mad_t *p_sa_mad; @@ -363,7 +358,7 @@ __osm_sa_mad_ctrl_rcv_callback(IN osm_madw_t * p_madw, #endif case IB_MAD_METHOD_SET: case IB_MAD_METHOD_DELETE: - __osm_sa_mad_ctrl_process(p_ctrl, p_madw); + sa_mad_ctrl_process(p_ctrl, p_madw); break; default: @@ -388,9 +383,9 @@ Exit: * SEE ALSO *********/ -/****f* opensm: SA/__osm_sa_mad_ctrl_send_err_callback +/****f* opensm: SA/sa_mad_ctrl_send_err_callback * NAME - * __osm_sa_mad_ctrl_send_err_callback + * sa_mad_ctrl_send_err_callback * * DESCRIPTION * This is the callback from the transport layer for send errors @@ -399,8 +394,7 @@ Exit: * SYNOPSIS */ static void -__osm_sa_mad_ctrl_send_err_callback(IN void *bind_context, - IN osm_madw_t * p_madw) +sa_mad_ctrl_send_err_callback(IN void *bind_context, IN osm_madw_t * p_madw) { osm_sa_mad_ctrl_t *p_ctrl = (osm_sa_mad_ctrl_t *) bind_context; cl_status_t status; @@ -425,7 +419,7 @@ __osm_sa_mad_ctrl_send_err_callback(IN void *bind_context, osm_dump_sa_mad(p_ctrl->p_log, osm_madw_get_sa_mad_ptr(p_madw), OSM_LOG_ERROR); - /* __osm_sm_mad_ctrl_update_wire_stats( p_ctrl ); */ + /* sm_mad_ctrl_update_wire_stats( p_ctrl ); */ if (osm_madw_get_err_msg(p_madw) != CL_DISP_MSGID_NONE) { OSM_LOG(p_ctrl->p_log, OSM_LOG_DEBUG, @@ -433,10 +427,8 @@ __osm_sa_mad_ctrl_send_err_callback(IN void *bind_context, osm_get_disp_msg_str(osm_madw_get_err_msg(p_madw))); status = cl_disp_post(p_ctrl->h_disp, - osm_madw_get_err_msg(p_madw), - p_madw, - __osm_sa_mad_ctrl_disp_done_callback, - p_ctrl); + osm_madw_get_err_msg(p_madw), p_madw, + sa_mad_ctrl_disp_done_callback, p_ctrl); if (status != CL_SUCCESS) { OSM_LOG(p_ctrl->p_log, OSM_LOG_ERROR, "ERR 1A07: " "Dispatcher post message failed (%s)\n", @@ -550,11 +542,10 @@ osm_sa_mad_ctrl_bind(IN osm_sa_mad_ctrl_t * const p_ctrl, OSM_LOG(p_ctrl->p_log, OSM_LOG_VERBOSE, "Binding to port GUID 0x%" PRIx64 "\n", cl_ntoh64(port_guid)); - p_ctrl->h_bind = osm_vendor_bind(p_ctrl->p_vendor, - &bind_info, + p_ctrl->h_bind = osm_vendor_bind(p_ctrl->p_vendor, &bind_info, p_ctrl->p_mad_pool, - __osm_sa_mad_ctrl_rcv_callback, - __osm_sa_mad_ctrl_send_err_callback, + sa_mad_ctrl_rcv_callback, + sa_mad_ctrl_send_err_callback, p_ctrl); if (p_ctrl->h_bind == OSM_BIND_INVALID_HANDLE) { diff --git a/opensm/opensm/osm_sa_mcmember_record.c b/opensm/opensm/osm_sa_mcmember_record.c index b586942..f87df31 100644 --- a/opensm/opensm/osm_sa_mcmember_record.c +++ b/opensm/opensm/osm_sa_mcmember_record.c @@ -228,9 +228,8 @@ __check_create_comp_mask(ib_net64_t comp_mask, Generate the response MAD **********************************************************************/ static void -__osm_mcmr_rcv_respond(IN osm_sa_t * sa, - IN osm_madw_t * const p_madw, - IN ib_member_rec_t * p_mcmember_rec) +mcmr_rcv_respond(IN osm_sa_t * sa, IN osm_madw_t * const p_madw, + IN ib_member_rec_t * p_mcmember_rec) { cl_qlist_t rec_list; osm_mcmr_item_t *item; @@ -1024,8 +1023,7 @@ osm_mcmr_rcv_find_or_create_new_mgrp(IN osm_sa_t * sa, Process a request for leaving the group **********************************************************************/ static void -__osm_mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, - IN osm_madw_t * const p_madw) +mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, IN osm_madw_t * const p_madw) { osm_mgrp_t *p_mgrp; ib_sa_mad_t *p_sa_mad; @@ -1098,8 +1096,7 @@ __osm_mcmr_rcv_leave_mgrp(IN osm_sa_t * sa, OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B09: " "osm_sm_mcgrp_leave failed\n"); - /* Send an SA response */ - __osm_mcmr_rcv_respond(sa, p_madw, &mcmember_rec); + mcmr_rcv_respond(sa, p_madw, &mcmember_rec); Exit: OSM_LOG_EXIT(sa->p_log); @@ -1109,7 +1106,7 @@ Exit: Handle a join (or create) request **********************************************************************/ static void -__osm_mcmr_rcv_join_mgrp(IN osm_sa_t * sa, IN osm_madw_t * const p_madw) +mcmr_rcv_join_mgrp(IN osm_sa_t * sa, IN osm_madw_t * const p_madw) { osm_mgrp_t *p_mgrp = NULL; ib_api_status_t status; @@ -1351,7 +1348,7 @@ __osm_mcmr_rcv_join_mgrp(IN osm_sa_t * sa, IN osm_madw_t * const p_madw) if (osm_log_is_active(sa->p_log, OSM_LOG_DEBUG)) osm_dump_mc_record(sa->p_log, &mcmember_rec, OSM_LOG_DEBUG); - __osm_mcmr_rcv_respond(sa, p_madw, &mcmember_rec); + mcmr_rcv_respond(sa, p_madw, &mcmember_rec); Exit: OSM_LOG_EXIT(sa->p_log); @@ -1361,9 +1358,8 @@ Exit: Add a patched multicast group to the results list **********************************************************************/ static ib_api_status_t -__osm_mcmr_rcv_new_mcmr(IN osm_sa_t * sa, - IN const ib_member_rec_t * p_rcvd_rec, - IN cl_qlist_t * const p_list) +mcmr_rcv_new_mcmr(IN osm_sa_t * sa, IN const ib_member_rec_t * p_rcvd_rec, + IN cl_qlist_t * const p_list) { osm_mcmr_item_t *p_rec_item; ib_api_status_t status = IB_SUCCESS; @@ -1538,7 +1534,7 @@ static void mcmr_by_comp_mask(osm_sa_t *sa, const ib_member_rec_t *p_rcvd_rec, match_rec.proxy_join = (uint8_t) (p_mcm_port->proxy_join); - __osm_mcmr_rcv_new_mcmr(sa, &match_rec, list); + mcmr_rcv_new_mcmr(sa, &match_rec, list); } p_item = cl_qmap_next(p_item); } @@ -1555,7 +1551,7 @@ static void mcmr_by_comp_mask(osm_sa_t *sa, const ib_member_rec_t *p_rcvd_rec, memcpy(&(match_rec.port_gid), &port_gid, sizeof(ib_gid_t)); match_rec.proxy_join = (uint8_t) proxy_join; - __osm_mcmr_rcv_new_mcmr(sa, &match_rec, list); + mcmr_rcv_new_mcmr(sa, &match_rec, list); } Exit: @@ -1566,8 +1562,7 @@ Exit: Handle a query request **********************************************************************/ static void -__osm_mcmr_query_mgrp(IN osm_sa_t * sa, - IN osm_madw_t * const p_madw) +mcmr_query_mgrp(IN osm_sa_t * sa, IN osm_madw_t * const p_madw) { const ib_sa_mad_t *p_rcvd_mad; const ib_member_rec_t *p_rcvd_rec; @@ -1687,7 +1682,7 @@ void osm_mcmr_rcv_process(IN void *context, IN void *data) /* * Join or Create Multicast Group */ - __osm_mcmr_rcv_join_mgrp(sa, p_madw); + mcmr_rcv_join_mgrp(sa, p_madw); break; case IB_MAD_METHOD_DELETE: if (!__check_join_comp_mask(p_sa_mad->comp_mask)) { @@ -1705,14 +1700,14 @@ void osm_mcmr_rcv_process(IN void *context, IN void *data) /* * Leave Multicast Group */ - __osm_mcmr_rcv_leave_mgrp(sa, p_madw); + mcmr_rcv_leave_mgrp(sa, p_madw); break; case IB_MAD_METHOD_GET: case IB_MAD_METHOD_GETTABLE: /* * Querying a Multicast Group */ - __osm_mcmr_query_mgrp(sa, p_madw); + mcmr_query_mgrp(sa, p_madw); break; default: OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1B21: " diff --git a/opensm/opensm/osm_sa_mft_record.c b/opensm/opensm/osm_sa_mft_record.c index bda9aee..0337853 100644 --- a/opensm/opensm/osm_sa_mft_record.c +++ b/opensm/opensm/osm_sa_mft_record.c @@ -70,11 +70,9 @@ typedef struct osm_mftr_search_ctxt { /********************************************************************** **********************************************************************/ static ib_api_status_t -__osm_mftr_rcv_new_mftr(IN osm_sa_t * sa, - IN osm_switch_t * const p_sw, - IN cl_qlist_t * const p_list, - IN ib_net16_t const lid, - IN uint16_t const block, IN uint8_t const position) +mftr_rcv_new_mftr(IN osm_sa_t * sa, IN osm_switch_t * const p_sw, + IN cl_qlist_t * const p_list, IN ib_net16_t const lid, + IN uint16_t const block, IN uint8_t const position) { osm_mftr_item_t *p_rec_item; ib_api_status_t status = IB_SUCCESS; @@ -117,8 +115,7 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_mftr_rcv_by_comp_mask(IN cl_map_item_t * const p_map_item, - IN void *context) +mftr_rcv_by_comp_mask(IN cl_map_item_t * const p_map_item, IN void *context) { const osm_mftr_search_ctxt_t *const p_ctxt = (osm_mftr_search_ctxt_t *) context; @@ -208,9 +205,9 @@ __osm_mftr_rcv_by_comp_mask(IN cl_map_item_t * const p_map_item, for (block = min_block; block <= max_block; block++) for (position = min_position; position <= max_position; position++) - __osm_mftr_rcv_new_mftr(sa, p_sw, p_ctxt->p_list, - osm_port_get_base_lid(p_port), - block, position); + mftr_rcv_new_mftr(sa, p_sw, p_ctxt->p_list, + osm_port_get_base_lid(p_port), + block, position); } /********************************************************************** @@ -268,8 +265,8 @@ void osm_mftr_rcv_process(IN void *ctx, IN void *data) cl_plock_acquire(sa->p_lock); /* Go over all switches */ - cl_qmap_apply_func(&sa->p_subn->sw_guid_tbl, - __osm_mftr_rcv_by_comp_mask, &context); + cl_qmap_apply_func(&sa->p_subn->sw_guid_tbl, mftr_rcv_by_comp_mask, + &context); cl_plock_release(sa->p_lock); diff --git a/opensm/opensm/osm_sa_multipath_record.c b/opensm/opensm/osm_sa_multipath_record.c index b9164be..0fa7193 100644 --- a/opensm/opensm/osm_sa_multipath_record.c +++ b/opensm/opensm/osm_sa_multipath_record.c @@ -84,7 +84,7 @@ typedef struct osm_path_parms { /********************************************************************** **********************************************************************/ static inline boolean_t -__osm_sa_multipath_rec_is_tavor_port(IN const osm_port_t * const p_port) +sa_multipath_rec_is_tavor_port(IN const osm_port_t * const p_port) { osm_node_t const *p_node; ib_net32_t vend_id; @@ -102,19 +102,16 @@ __osm_sa_multipath_rec_is_tavor_port(IN const osm_port_t * const p_port) /********************************************************************** **********************************************************************/ static boolean_t -__osm_sa_multipath_rec_apply_tavor_mtu_limit(IN const ib_multipath_rec_t * - const p_mpr, - IN const osm_port_t * - const p_src_port, - IN const osm_port_t * - const p_dest_port, - IN const ib_net64_t comp_mask) +sa_multipath_rec_apply_tavor_mtu_limit(IN const ib_multipath_rec_t * const p_mpr, + IN const osm_port_t * const p_src_port, + IN const osm_port_t * const p_dest_port, + IN const ib_net64_t comp_mask) { uint8_t required_mtu; /* only if at least one of the ports is a Tavor device */ - if (!__osm_sa_multipath_rec_is_tavor_port(p_src_port) && - !__osm_sa_multipath_rec_is_tavor_port(p_dest_port)) + if (!sa_multipath_rec_is_tavor_port(p_src_port) && + !sa_multipath_rec_is_tavor_port(p_dest_port)) return (FALSE); /* @@ -157,13 +154,13 @@ __osm_sa_multipath_rec_apply_tavor_mtu_limit(IN const ib_multipath_rec_t * /********************************************************************** **********************************************************************/ static ib_api_status_t -__osm_mpr_rcv_get_path_parms(IN osm_sa_t * sa, - IN const ib_multipath_rec_t * const p_mpr, - IN const osm_port_t * const p_src_port, - IN const osm_port_t * const p_dest_port, - IN const uint16_t dest_lid_ho, - IN const ib_net64_t comp_mask, - OUT osm_path_parms_t * const p_parms) +mpr_rcv_get_path_parms(IN osm_sa_t * sa, + IN const ib_multipath_rec_t * const p_mpr, + IN const osm_port_t * const p_src_port, + IN const osm_port_t * const p_dest_port, + IN const uint16_t dest_lid_ho, + IN const ib_net64_t comp_mask, + OUT osm_path_parms_t * const p_parms) { const osm_node_t *p_node; const osm_physp_t *p_physp; @@ -207,9 +204,8 @@ __osm_mpr_rcv_get_path_parms(IN osm_sa_t * sa, port MTU with 1K. */ if (sa->p_subn->opt.enable_quirks && - __osm_sa_multipath_rec_apply_tavor_mtu_limit(p_mpr, p_src_port, - p_dest_port, - comp_mask)) + sa_multipath_rec_apply_tavor_mtu_limit(p_mpr, p_src_port, + p_dest_port, comp_mask)) if (mtu > IB_MTU_LEN_1024) { mtu = IB_MTU_LEN_1024; OSM_LOG(sa->p_log, OSM_LOG_DEBUG, @@ -744,14 +740,12 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_mpr_rcv_build_pr(IN osm_sa_t * sa, - IN const osm_port_t * const p_src_port, - IN const osm_port_t * const p_dest_port, - IN const uint16_t src_lid_ho, - IN const uint16_t dest_lid_ho, - IN const uint8_t preference, - IN const osm_path_parms_t * const p_parms, - OUT ib_path_rec_t * const p_pr) +mpr_rcv_build_pr(IN osm_sa_t * sa, IN const osm_port_t * const p_src_port, + IN const osm_port_t * const p_dest_port, + IN const uint16_t src_lid_ho, IN const uint16_t dest_lid_ho, + IN const uint8_t preference, + IN const osm_path_parms_t * const p_parms, + OUT ib_path_rec_t * const p_pr) { const osm_physp_t *p_src_physp; const osm_physp_t *p_dest_physp; @@ -797,14 +791,14 @@ __osm_mpr_rcv_build_pr(IN osm_sa_t * sa, /********************************************************************** **********************************************************************/ static osm_mpr_item_t * -__osm_mpr_rcv_get_lid_pair_path(IN osm_sa_t * sa, - IN const ib_multipath_rec_t * const p_mpr, - IN const osm_port_t * const p_src_port, - IN const osm_port_t * const p_dest_port, - IN const uint16_t src_lid_ho, - IN const uint16_t dest_lid_ho, - IN const ib_net64_t comp_mask, - IN const uint8_t preference) +mpr_rcv_get_lid_pair_path(IN osm_sa_t * sa, + IN const ib_multipath_rec_t * const p_mpr, + IN const osm_port_t * const p_src_port, + IN const osm_port_t * const p_dest_port, + IN const uint16_t src_lid_ho, + IN const uint16_t dest_lid_ho, + IN const ib_net64_t comp_mask, + IN const uint8_t preference) { osm_path_parms_t path_parms; osm_path_parms_t rev_path_parms; @@ -824,9 +818,8 @@ __osm_mpr_rcv_get_lid_pair_path(IN osm_sa_t * sa, } memset(p_pr_item, 0, sizeof(*p_pr_item)); - status = __osm_mpr_rcv_get_path_parms(sa, p_mpr, p_src_port, - p_dest_port, dest_lid_ho, - comp_mask, &path_parms); + status = mpr_rcv_get_path_parms(sa, p_mpr, p_src_port, p_dest_port, + dest_lid_ho, comp_mask, &path_parms); if (status != IB_SUCCESS) { free(p_pr_item); @@ -835,10 +828,9 @@ __osm_mpr_rcv_get_lid_pair_path(IN osm_sa_t * sa, } /* now try the reversible path */ - rev_path_status = - __osm_mpr_rcv_get_path_parms(sa, p_mpr, p_dest_port, p_src_port, - src_lid_ho, comp_mask, - &rev_path_parms); + rev_path_status = mpr_rcv_get_path_parms(sa, p_mpr, p_dest_port, + p_src_port, src_lid_ho, + comp_mask, &rev_path_parms); path_parms.reversible = (rev_path_status == IB_SUCCESS); /* did we get a Reversible Path compmask ? */ @@ -862,9 +854,8 @@ __osm_mpr_rcv_get_lid_pair_path(IN osm_sa_t * sa, p_pr_item->p_dest_port = p_dest_port; p_pr_item->hops = path_parms.hops; - __osm_mpr_rcv_build_pr(sa, p_src_port, p_dest_port, src_lid_ho, - dest_lid_ho, preference, &path_parms, - &p_pr_item->path_rec); + mpr_rcv_build_pr(sa, p_src_port, p_dest_port, src_lid_ho, dest_lid_ho, + preference, &path_parms, &p_pr_item->path_rec); Exit: OSM_LOG_EXIT(sa->p_log); @@ -874,14 +865,14 @@ Exit: /********************************************************************** **********************************************************************/ static uint32_t -__osm_mpr_rcv_get_port_pair_paths(IN osm_sa_t * sa, - IN const ib_multipath_rec_t * const p_mpr, - IN const osm_port_t * const p_req_port, - IN const osm_port_t * const p_src_port, - IN const osm_port_t * const p_dest_port, - IN const uint32_t rem_paths, - IN const ib_net64_t comp_mask, - IN cl_qlist_t * const p_list) +mpr_rcv_get_port_pair_paths(IN osm_sa_t * sa, + IN const ib_multipath_rec_t * const p_mpr, + IN const osm_port_t * const p_req_port, + IN const osm_port_t * const p_src_port, + IN const osm_port_t * const p_dest_port, + IN const uint32_t rem_paths, + IN const ib_net64_t comp_mask, + IN cl_qlist_t * const p_list) { osm_mpr_item_t *p_pr_item; uint16_t src_lid_min_ho; @@ -977,13 +968,10 @@ __osm_mpr_rcv_get_port_pair_paths(IN osm_sa_t * sa, /* These paths are "fully redundant" */ - p_pr_item = __osm_mpr_rcv_get_lid_pair_path(sa, p_mpr, - p_src_port, - p_dest_port, - src_lid_ho, - dest_lid_ho, - comp_mask, - preference); + p_pr_item = mpr_rcv_get_lid_pair_path(sa, p_mpr, p_src_port, + p_dest_port, src_lid_ho, + dest_lid_ho, comp_mask, + preference); if (p_pr_item) { cl_qlist_insert_tail(p_list, &p_pr_item->list_item); @@ -1043,13 +1031,10 @@ __osm_mpr_rcv_get_port_pair_paths(IN osm_sa_t * sa, if (src_offset == dest_offset) continue; /* already reported */ - p_pr_item = __osm_mpr_rcv_get_lid_pair_path(sa, p_mpr, - p_src_port, - p_dest_port, - src_lid_ho, - dest_lid_ho, - comp_mask, - preference); + p_pr_item = mpr_rcv_get_lid_pair_path(sa, p_mpr, p_src_port, + p_dest_port, src_lid_ho, + dest_lid_ho, comp_mask, + preference); if (p_pr_item) { cl_qlist_insert_tail(p_list, &p_pr_item->list_item); @@ -1068,13 +1053,13 @@ Exit: /********************************************************************** **********************************************************************/ static osm_mpr_item_t * -__osm_mpr_rcv_get_apm_port_pair_paths(IN osm_sa_t * sa, - IN const ib_multipath_rec_t * const p_mpr, - IN const osm_port_t * const p_src_port, - IN const osm_port_t * const p_dest_port, - IN int base_offs, - IN const ib_net64_t comp_mask, - IN cl_qlist_t * const p_list) +mpr_rcv_get_apm_port_pair_paths(IN osm_sa_t * sa, + IN const ib_multipath_rec_t * const p_mpr, + IN const osm_port_t * const p_src_port, + IN const osm_port_t * const p_dest_port, + IN int base_offs, + IN const ib_net64_t comp_mask, + IN cl_qlist_t * const p_list) { osm_mpr_item_t *p_pr_item = 0; uint16_t src_lid_min_ho; @@ -1118,12 +1103,9 @@ __osm_mpr_rcv_get_apm_port_pair_paths(IN osm_sa_t * sa, /* These paths are "fully redundant" */ - p_pr_item = __osm_mpr_rcv_get_lid_pair_path(sa, p_mpr, - p_src_port, - p_dest_port, - src_lid_ho, - dest_lid_ho, - comp_mask, 0); + p_pr_item = mpr_rcv_get_lid_pair_path(sa, p_mpr, p_src_port, + p_dest_port, src_lid_ho, + dest_lid_ho, comp_mask, 0); if (p_pr_item) { OSM_LOG(sa->p_log, OSM_LOG_DEBUG, @@ -1146,9 +1128,8 @@ __osm_mpr_rcv_get_apm_port_pair_paths(IN osm_sa_t * sa, /********************************************************************** **********************************************************************/ static ib_net16_t -__osm_mpr_rcv_get_gids(IN osm_sa_t * sa, - IN const ib_gid_t * gids, - IN int ngids, IN int is_sgid, OUT osm_port_t ** pp_port) +mpr_rcv_get_gids(IN osm_sa_t * sa, IN const ib_gid_t * gids, IN int ngids, + IN int is_sgid, OUT osm_port_t ** pp_port) { osm_port_t *p_port; ib_net16_t ib_status = IB_SUCCESS; @@ -1206,10 +1187,9 @@ Exit: /********************************************************************** **********************************************************************/ static ib_net16_t -__osm_mpr_rcv_get_end_points(IN osm_sa_t * sa, - IN const osm_madw_t * const p_madw, - OUT osm_port_t ** pp_ports, - OUT int *nsrc, OUT int *ndest) +mpr_rcv_get_end_points(IN osm_sa_t * sa, IN const osm_madw_t * const p_madw, + OUT osm_port_t ** pp_ports, OUT int *nsrc, + OUT int *ndest) { const ib_multipath_rec_t *p_mpr; const ib_sa_mad_t *p_sa_mad; @@ -1239,8 +1219,7 @@ __osm_mpr_rcv_get_end_points(IN osm_sa_t * sa, *nsrc = p_mpr->sgid_count; if (*nsrc > IB_MULTIPATH_MAX_GIDS) *nsrc = IB_MULTIPATH_MAX_GIDS; - sa_status = - __osm_mpr_rcv_get_gids(sa, gids, *nsrc, 1, pp_ports); + sa_status = mpr_rcv_get_gids(sa, gids, *nsrc, 1, pp_ports); if (sa_status != IB_SUCCESS) goto Exit; } @@ -1250,8 +1229,8 @@ __osm_mpr_rcv_get_end_points(IN osm_sa_t * sa, if (*ndest + *nsrc > IB_MULTIPATH_MAX_GIDS) *ndest = IB_MULTIPATH_MAX_GIDS - *nsrc; sa_status = - __osm_mpr_rcv_get_gids(sa, gids + *nsrc, *ndest, 0, - pp_ports + *nsrc); + mpr_rcv_get_gids(sa, gids + *nsrc, *ndest, 0, + pp_ports + *nsrc); } Exit: @@ -1259,18 +1238,18 @@ Exit: return (sa_status); } -#define __hash_lids(a, b, lmc) \ +#define hash_lids(a, b, lmc) \ (((((a) >> (lmc)) << 4) | ((b) >> (lmc))) % 103) /********************************************************************** **********************************************************************/ static void -__osm_mpr_rcv_get_apm_paths(IN osm_sa_t * sa, - IN const ib_multipath_rec_t * const p_mpr, - IN const osm_port_t * const p_req_port, - IN osm_port_t ** _pp_ports, - IN const ib_net64_t comp_mask, - IN cl_qlist_t * const p_list) +mpr_rcv_get_apm_paths(IN osm_sa_t * sa, + IN const ib_multipath_rec_t * const p_mpr, + IN const osm_port_t * const p_req_port, + IN osm_port_t ** _pp_ports, + IN const ib_net64_t comp_mask, + IN cl_qlist_t * const p_list) { osm_port_t *pp_ports[4]; osm_mpr_item_t *matrix[2][2]; @@ -1287,9 +1266,9 @@ __osm_mpr_rcv_get_apm_paths(IN osm_sa_t * sa, * 2. keep consistent paths no matter of direction and order of ports * 3. distibute the lid offsets to balance the load * So, we sort the ports (within the srcs, and within the dests), - * hash the lids of S0, D0 (after the sort), and call __osm_mpr_rcv_get_apm_port_pair_paths + * hash the lids of S0, D0 (after the sort), and call mpr_rcv_get_apm_port_pair_paths * with base_lid for S0, D0 and base_lid + 1 for S1, D1. This way we will get - * always the same offsets - order indepentent, and make sure different spines are used. + * always the same offsets - order independent, and make sure different spines are used. * Note that the diagonals on a Clos have the same number of hops, so it doesn't * really matter which diagonal we use. */ @@ -1312,25 +1291,21 @@ __osm_mpr_rcv_get_apm_paths(IN osm_sa_t * sa, dest_lid_ho = osm_port_get_base_lid(pp_ports[2]); base_offs = src_lid_ho < dest_lid_ho ? - __hash_lids(src_lid_ho, dest_lid_ho, sa->p_subn->opt.lmc) : - __hash_lids(dest_lid_ho, src_lid_ho, sa->p_subn->opt.lmc); + hash_lids(src_lid_ho, dest_lid_ho, sa->p_subn->opt.lmc) : + hash_lids(dest_lid_ho, src_lid_ho, sa->p_subn->opt.lmc); matrix[0][0] = - __osm_mpr_rcv_get_apm_port_pair_paths(sa, p_mpr, pp_ports[0], - pp_ports[2], base_offs, - comp_mask, p_list); + mpr_rcv_get_apm_port_pair_paths(sa, p_mpr, pp_ports[0], pp_ports[2], + base_offs, comp_mask, p_list); matrix[0][1] = - __osm_mpr_rcv_get_apm_port_pair_paths(sa, p_mpr, pp_ports[0], - pp_ports[3], base_offs, - comp_mask, p_list); + mpr_rcv_get_apm_port_pair_paths(sa, p_mpr, pp_ports[0], pp_ports[3], + base_offs, comp_mask, p_list); matrix[1][0] = - __osm_mpr_rcv_get_apm_port_pair_paths(sa, p_mpr, pp_ports[1], - pp_ports[2], base_offs + 1, - comp_mask, p_list); + mpr_rcv_get_apm_port_pair_paths(sa, p_mpr, pp_ports[1], pp_ports[2], + base_offs + 1, comp_mask, p_list); matrix[1][1] = - __osm_mpr_rcv_get_apm_port_pair_paths(sa, p_mpr, pp_ports[1], - pp_ports[3], base_offs + 1, - comp_mask, p_list); + mpr_rcv_get_apm_port_pair_paths(sa, p_mpr, pp_ports[1], pp_ports[3], + base_offs + 1, comp_mask, p_list); OSM_LOG(sa->p_log, OSM_LOG_DEBUG, "APM matrix:\n" "\t{0,0} 0x%X->0x%X (%d)\t| {0,1} 0x%X->0x%X (%d)\n" @@ -1385,14 +1360,13 @@ __osm_mpr_rcv_get_apm_paths(IN osm_sa_t * sa, /********************************************************************** **********************************************************************/ static void -__osm_mpr_rcv_process_pairs(IN osm_sa_t * sa, - IN const ib_multipath_rec_t * const p_mpr, - IN osm_port_t * const p_req_port, - IN osm_port_t ** pp_ports, - IN const int nsrc, - IN const int ndest, - IN const ib_net64_t comp_mask, - IN cl_qlist_t * const p_list) +mpr_rcv_process_pairs(IN osm_sa_t * sa, + IN const ib_multipath_rec_t * const p_mpr, + IN osm_port_t * const p_req_port, + IN osm_port_t ** pp_ports, + IN const int nsrc, IN const int ndest, + IN const ib_net64_t comp_mask, + IN cl_qlist_t * const p_list) { osm_port_t **pp_src_port, **pp_es; osm_port_t **pp_dest_port, **pp_ed; @@ -1410,14 +1384,11 @@ __osm_mpr_rcv_process_pairs(IN osm_sa_t * sa, for (pp_dest_port = pp_es, pp_ed = pp_es + ndest; pp_dest_port < pp_ed; pp_dest_port++) { num_paths = - __osm_mpr_rcv_get_port_pair_paths(sa, p_mpr, - p_req_port, - *pp_src_port, - *pp_dest_port, - max_paths - - total_paths, - comp_mask, - p_list); + mpr_rcv_get_port_pair_paths(sa, p_mpr, p_req_port, + *pp_src_port, + *pp_dest_port, + max_paths - total_paths, + comp_mask, p_list); total_paths += num_paths; OSM_LOG(sa->p_log, OSM_LOG_DEBUG, "%d paths %d total paths %d max paths\n", @@ -1492,13 +1463,12 @@ void osm_mpr_rcv_process(IN void *context, IN void *data) */ cl_plock_acquire(sa->p_lock); - sa_status = __osm_mpr_rcv_get_end_points(sa, p_madw, pp_ports, - &nsrc, &ndest); + sa_status = mpr_rcv_get_end_points(sa, p_madw, pp_ports, &nsrc, &ndest); if (sa_status != IB_SA_MAD_STATUS_SUCCESS || !nsrc || !ndest) { if (sa_status == IB_SA_MAD_STATUS_SUCCESS && (!nsrc || !ndest)) OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 4512: " - "__osm_mpr_rcv_get_end_points failed, not enough GIDs " + "mpr_rcv_get_end_points failed, not enough GIDs " "(nsrc %d ndest %d)\n", nsrc, ndest); cl_plock_release(sa->p_lock); if (sa_status == IB_SA_MAD_STATUS_SUCCESS) @@ -1511,13 +1481,11 @@ void osm_mpr_rcv_process(IN void *context, IN void *data) /* APM request */ if (nsrc == 2 && ndest == 2 && (p_mpr->num_path & 0x7F) == 2) - __osm_mpr_rcv_get_apm_paths(sa, p_mpr, requester_port, - pp_ports, p_sa_mad->comp_mask, - &pr_list); + mpr_rcv_get_apm_paths(sa, p_mpr, requester_port, pp_ports, + p_sa_mad->comp_mask, &pr_list); else - __osm_mpr_rcv_process_pairs(sa, p_mpr, requester_port, - pp_ports, nsrc, ndest, - p_sa_mad->comp_mask, &pr_list); + mpr_rcv_process_pairs(sa, p_mpr, requester_port, pp_ports, + nsrc, ndest, p_sa_mad->comp_mask, &pr_list); cl_plock_release(sa->p_lock); diff --git a/opensm/opensm/osm_sa_node_record.c b/opensm/opensm/osm_sa_node_record.c index 4df2c91..8afc252 100644 --- a/opensm/opensm/osm_sa_node_record.c +++ b/opensm/opensm/osm_sa_node_record.c @@ -70,10 +70,9 @@ typedef struct osm_nr_search_ctxt { /********************************************************************** **********************************************************************/ static ib_api_status_t -__osm_nr_rcv_new_nr(IN osm_sa_t * sa, - IN const osm_node_t * const p_node, - IN cl_qlist_t * const p_list, - IN ib_net64_t const port_guid, IN ib_net16_t const lid) +nr_rcv_new_nr(IN osm_sa_t * sa, IN const osm_node_t * const p_node, + IN cl_qlist_t * const p_list, IN ib_net64_t const port_guid, + IN ib_net16_t const lid) { osm_nr_item_t *p_rec_item; ib_api_status_t status = IB_SUCCESS; @@ -112,12 +111,11 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_nr_rcv_create_nr(IN osm_sa_t * sa, - IN osm_node_t * const p_node, - IN cl_qlist_t * const p_list, - IN ib_net64_t const match_port_guid, - IN ib_net16_t const match_lid, - IN const osm_physp_t * const p_req_physp) +nr_rcv_create_nr(IN osm_sa_t * sa, IN osm_node_t * const p_node, + IN cl_qlist_t * const p_list, + IN ib_net64_t const match_port_guid, + IN ib_net16_t const match_lid, + IN const osm_physp_t * const p_req_physp) { const osm_physp_t *p_physp; uint8_t port_num; @@ -179,7 +177,7 @@ __osm_nr_rcv_create_nr(IN osm_sa_t * sa, continue; } - __osm_nr_rcv_new_nr(sa, p_node, p_list, port_guid, base_lid); + nr_rcv_new_nr(sa, p_node, p_list, port_guid, base_lid); } @@ -189,7 +187,7 @@ __osm_nr_rcv_create_nr(IN osm_sa_t * sa, /********************************************************************** **********************************************************************/ static void -__osm_nr_rcv_by_comp_mask(IN cl_map_item_t * const p_map_item, IN void *context) +nr_rcv_by_comp_mask(IN cl_map_item_t * const p_map_item, IN void *context) { const osm_nr_search_ctxt_t *const p_ctxt = (osm_nr_search_ctxt_t *) context; @@ -280,8 +278,8 @@ __osm_nr_rcv_by_comp_mask(IN cl_map_item_t * const p_map_item, IN void *context) goto Exit; } - __osm_nr_rcv_create_nr(sa, p_node, p_ctxt->p_list, - match_port_guid, match_lid, p_req_physp); + nr_rcv_create_nr(sa, p_node, p_ctxt->p_list, match_port_guid, + match_lid, p_req_physp); Exit: OSM_LOG_EXIT(p_ctxt->sa->p_log); @@ -343,8 +341,8 @@ void osm_nr_rcv_process(IN void *ctx, IN void *data) cl_plock_acquire(sa->p_lock); - cl_qmap_apply_func(&sa->p_subn->node_guid_tbl, - __osm_nr_rcv_by_comp_mask, &context); + cl_qmap_apply_func(&sa->p_subn->node_guid_tbl, nr_rcv_by_comp_mask, + &context); cl_plock_release(sa->p_lock); diff --git a/opensm/opensm/osm_sa_path_record.c b/opensm/opensm/osm_sa_path_record.c index 978c1a5..fa099d3 100644 --- a/opensm/opensm/osm_sa_path_record.c +++ b/opensm/opensm/osm_sa_path_record.c @@ -87,7 +87,7 @@ typedef struct osm_path_parms { /********************************************************************** **********************************************************************/ static inline boolean_t -__osm_sa_path_rec_is_tavor_port(IN const osm_port_t * const p_port) +sa_path_rec_is_tavor_port(IN const osm_port_t * const p_port) { osm_node_t const *p_node; ib_net32_t vend_id; @@ -105,16 +105,16 @@ __osm_sa_path_rec_is_tavor_port(IN const osm_port_t * const p_port) /********************************************************************** **********************************************************************/ static boolean_t -__osm_sa_path_rec_apply_tavor_mtu_limit(IN const ib_path_rec_t * const p_pr, - IN const osm_port_t * const p_src_port, - IN const osm_port_t * const p_dest_port, - IN const ib_net64_t comp_mask) +sa_path_rec_apply_tavor_mtu_limit(IN const ib_path_rec_t * const p_pr, + IN const osm_port_t * const p_src_port, + IN const osm_port_t * const p_dest_port, + IN const ib_net64_t comp_mask) { uint8_t required_mtu; /* only if at least one of the ports is a Tavor device */ - if (!__osm_sa_path_rec_is_tavor_port(p_src_port) && - !__osm_sa_path_rec_is_tavor_port(p_dest_port)) + if (!sa_path_rec_is_tavor_port(p_src_port) && + !sa_path_rec_is_tavor_port(p_dest_port)) return (FALSE); /* @@ -157,13 +157,12 @@ __osm_sa_path_rec_apply_tavor_mtu_limit(IN const ib_path_rec_t * const p_pr, /********************************************************************** **********************************************************************/ static ib_api_status_t -__osm_pr_rcv_get_path_parms(IN osm_sa_t * sa, - IN const ib_path_rec_t * const p_pr, - IN const osm_port_t * const p_src_port, - IN const osm_port_t * const p_dest_port, - IN const uint16_t dest_lid_ho, - IN const ib_net64_t comp_mask, - OUT osm_path_parms_t * const p_parms) +pr_rcv_get_path_parms(IN osm_sa_t * sa, IN const ib_path_rec_t * const p_pr, + IN const osm_port_t * const p_src_port, + IN const osm_port_t * const p_dest_port, + IN const uint16_t dest_lid_ho, + IN const ib_net64_t comp_mask, + OUT osm_path_parms_t * const p_parms) { const osm_node_t *p_node; const osm_physp_t *p_physp; @@ -209,8 +208,8 @@ __osm_pr_rcv_get_path_parms(IN osm_sa_t * sa, port MTU with 1K. */ if (sa->p_subn->opt.enable_quirks && - __osm_sa_path_rec_apply_tavor_mtu_limit(p_pr, p_src_port, - p_dest_port, comp_mask)) + sa_path_rec_apply_tavor_mtu_limit(p_pr, p_src_port, p_dest_port, + comp_mask)) if (mtu > IB_MTU_LEN_1024) { mtu = IB_MTU_LEN_1024; OSM_LOG(sa->p_log, OSM_LOG_DEBUG, @@ -752,15 +751,13 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_pr_rcv_build_pr(IN osm_sa_t * sa, - IN const osm_port_t * const p_src_port, - IN const osm_port_t * const p_dest_port, - IN const ib_gid_t * const p_dgid, - IN const uint16_t src_lid_ho, - IN const uint16_t dest_lid_ho, - IN const uint8_t preference, - IN const osm_path_parms_t * const p_parms, - OUT ib_path_rec_t * const p_pr) +pr_rcv_build_pr(IN osm_sa_t * sa, IN const osm_port_t * const p_src_port, + IN const osm_port_t * const p_dest_port, + IN const ib_gid_t * const p_dgid, + IN const uint16_t src_lid_ho, IN const uint16_t dest_lid_ho, + IN const uint8_t preference, + IN const osm_path_parms_t * const p_parms, + OUT ib_path_rec_t * const p_pr) { const osm_physp_t *p_src_physp; const osm_physp_t *p_dest_physp; @@ -821,15 +818,14 @@ __osm_pr_rcv_build_pr(IN osm_sa_t * sa, /********************************************************************** **********************************************************************/ static osm_pr_item_t * -__osm_pr_rcv_get_lid_pair_path(IN osm_sa_t * sa, - IN const ib_path_rec_t * const p_pr, - IN const osm_port_t * const p_src_port, - IN const osm_port_t * const p_dest_port, - IN const ib_gid_t * const p_dgid, - IN const uint16_t src_lid_ho, - IN const uint16_t dest_lid_ho, - IN const ib_net64_t comp_mask, - IN const uint8_t preference) +pr_rcv_get_lid_pair_path(IN osm_sa_t * sa, IN const ib_path_rec_t * const p_pr, + IN const osm_port_t * const p_src_port, + IN const osm_port_t * const p_dest_port, + IN const ib_gid_t * const p_dgid, + IN const uint16_t src_lid_ho, + IN const uint16_t dest_lid_ho, + IN const ib_net64_t comp_mask, + IN const uint8_t preference) { osm_path_parms_t path_parms; osm_path_parms_t rev_path_parms; @@ -849,9 +845,8 @@ __osm_pr_rcv_get_lid_pair_path(IN osm_sa_t * sa, } memset(p_pr_item, 0, sizeof(*p_pr_item)); - status = __osm_pr_rcv_get_path_parms(sa, p_pr, p_src_port, - p_dest_port, dest_lid_ho, - comp_mask, &path_parms); + status = pr_rcv_get_path_parms(sa, p_pr, p_src_port, p_dest_port, + dest_lid_ho, comp_mask, &path_parms); if (status != IB_SUCCESS) { free(p_pr_item); @@ -860,10 +855,9 @@ __osm_pr_rcv_get_lid_pair_path(IN osm_sa_t * sa, } /* now try the reversible path */ - rev_path_status = __osm_pr_rcv_get_path_parms(sa, p_pr, p_dest_port, - p_src_port, src_lid_ho, - comp_mask, - &rev_path_parms); + rev_path_status = pr_rcv_get_path_parms(sa, p_pr, p_dest_port, + p_src_port, src_lid_ho, + comp_mask, &rev_path_parms); path_parms.reversible = (rev_path_status == IB_SUCCESS); /* did we get a Reversible Path compmask ? */ @@ -883,9 +877,9 @@ __osm_pr_rcv_get_lid_pair_path(IN osm_sa_t * sa, } } - __osm_pr_rcv_build_pr(sa, p_src_port, p_dest_port, p_dgid, - src_lid_ho, dest_lid_ho, preference, &path_parms, - &p_pr_item->path_rec); + pr_rcv_build_pr(sa, p_src_port, p_dest_port, p_dgid, src_lid_ho, + dest_lid_ho, preference, &path_parms, + &p_pr_item->path_rec); Exit: OSM_LOG_EXIT(sa->p_log); @@ -895,14 +889,13 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, - IN const osm_madw_t * const p_madw, - IN const osm_port_t * const p_req_port, - IN const osm_port_t * const p_src_port, - IN const osm_port_t * const p_dest_port, - IN const ib_gid_t * const p_dgid, - IN const ib_net64_t comp_mask, - IN cl_qlist_t * const p_list) +pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, IN const osm_madw_t * const p_madw, + IN const osm_port_t * const p_req_port, + IN const osm_port_t * const p_src_port, + IN const osm_port_t * const p_dest_port, + IN const ib_gid_t * const p_dgid, + IN const ib_net64_t comp_mask, + IN cl_qlist_t * const p_list) { const ib_path_rec_t *p_pr; const ib_sa_mad_t *p_sa_mad; @@ -1041,13 +1034,10 @@ __osm_pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, These paths are "fully redundant" */ - p_pr_item = __osm_pr_rcv_get_lid_pair_path(sa, p_pr, - p_src_port, - p_dest_port, p_dgid, - src_lid_ho, - dest_lid_ho, - comp_mask, - preference); + p_pr_item = pr_rcv_get_lid_pair_path(sa, p_pr, p_src_port, + p_dest_port, p_dgid, + src_lid_ho, dest_lid_ho, + comp_mask, preference); if (p_pr_item) { cl_qlist_insert_tail(p_list, &p_pr_item->list_item); @@ -1107,13 +1097,10 @@ __osm_pr_rcv_get_port_pair_paths(IN osm_sa_t * sa, if (src_offset == dest_offset) continue; /* already reported */ - p_pr_item = __osm_pr_rcv_get_lid_pair_path(sa, p_pr, - p_src_port, - p_dest_port, p_dgid, - src_lid_ho, - dest_lid_ho, - comp_mask, - preference); + p_pr_item = pr_rcv_get_lid_pair_path(sa, p_pr, p_src_port, + p_dest_port, p_dgid, + src_lid_ho, dest_lid_ho, + comp_mask, preference); if (p_pr_item) { cl_qlist_insert_tail(p_list, &p_pr_item->list_item); @@ -1128,11 +1115,10 @@ Exit: /********************************************************************** **********************************************************************/ static ib_net16_t -__osm_pr_rcv_get_end_points(IN osm_sa_t * sa, - IN const osm_madw_t * const p_madw, - OUT const osm_port_t ** const pp_src_port, - OUT const osm_port_t ** const pp_dest_port, - OUT ib_gid_t * const p_dgid) +pr_rcv_get_end_points(IN osm_sa_t * sa, IN const osm_madw_t * const p_madw, + OUT const osm_port_t ** const pp_src_port, + OUT const osm_port_t ** const pp_dest_port, + OUT ib_gid_t * const p_dgid) { const ib_path_rec_t *p_pr; const ib_sa_mad_t *p_sa_mad; @@ -1337,12 +1323,11 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_pr_rcv_process_world(IN osm_sa_t * sa, - IN const osm_madw_t * const p_madw, - IN const osm_port_t * const requester_port, - IN const ib_gid_t * const p_dgid, - IN const ib_net64_t comp_mask, - IN cl_qlist_t * const p_list) +pr_rcv_process_world(IN osm_sa_t * sa, IN const osm_madw_t * const p_madw, + IN const osm_port_t * const requester_port, + IN const ib_gid_t * const p_dgid, + IN const ib_net64_t comp_mask, + IN cl_qlist_t * const p_list) { const cl_qmap_t *p_tbl; const osm_port_t *p_dest_port; @@ -1364,11 +1349,9 @@ __osm_pr_rcv_process_world(IN osm_sa_t * sa, while (p_dest_port != (osm_port_t *) cl_qmap_end(p_tbl)) { p_src_port = (osm_port_t *) cl_qmap_head(p_tbl); while (p_src_port != (osm_port_t *) cl_qmap_end(p_tbl)) { - __osm_pr_rcv_get_port_pair_paths(sa, p_madw, - requester_port, - p_src_port, - p_dest_port, p_dgid, - comp_mask, p_list); + pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, + p_src_port, p_dest_port, + p_dgid, comp_mask, p_list); p_src_port = (osm_port_t *) cl_qmap_next(&p_src_port->map_item); @@ -1384,14 +1367,13 @@ __osm_pr_rcv_process_world(IN osm_sa_t * sa, /********************************************************************** **********************************************************************/ static void -__osm_pr_rcv_process_half(IN osm_sa_t * sa, - IN const osm_madw_t * const p_madw, - IN const osm_port_t * const requester_port, - IN const osm_port_t * const p_src_port, - IN const osm_port_t * const p_dest_port, - IN const ib_gid_t * const p_dgid, - IN const ib_net64_t comp_mask, - IN cl_qlist_t * const p_list) +pr_rcv_process_half(IN osm_sa_t * sa, IN const osm_madw_t * const p_madw, + IN const osm_port_t * const requester_port, + IN const osm_port_t * const p_src_port, + IN const osm_port_t * const p_dest_port, + IN const ib_gid_t * const p_dgid, + IN const ib_net64_t comp_mask, + IN cl_qlist_t * const p_list) { const cl_qmap_t *p_tbl; const osm_port_t *p_port; @@ -1411,11 +1393,9 @@ __osm_pr_rcv_process_half(IN osm_sa_t * sa, */ p_port = (osm_port_t *) cl_qmap_head(p_tbl); while (p_port != (osm_port_t *) cl_qmap_end(p_tbl)) { - __osm_pr_rcv_get_port_pair_paths(sa, p_madw, - requester_port, - p_src_port, p_port, - p_dgid, comp_mask, - p_list); + pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, + p_src_port, p_port, p_dgid, + comp_mask, p_list); p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item); } } else { @@ -1424,10 +1404,9 @@ __osm_pr_rcv_process_half(IN osm_sa_t * sa, */ p_port = (osm_port_t *) cl_qmap_head(p_tbl); while (p_port != (osm_port_t *) cl_qmap_end(p_tbl)) { - __osm_pr_rcv_get_port_pair_paths(sa, p_madw, - requester_port, p_port, - p_dest_port, p_dgid, - comp_mask, p_list); + pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, + p_port, p_dest_port, p_dgid, + comp_mask, p_list); p_port = (osm_port_t *) cl_qmap_next(&p_port->map_item); } } @@ -1438,20 +1417,18 @@ __osm_pr_rcv_process_half(IN osm_sa_t * sa, /********************************************************************** **********************************************************************/ static void -__osm_pr_rcv_process_pair(IN osm_sa_t * sa, - IN const osm_madw_t * const p_madw, - IN const osm_port_t * const requester_port, - IN const osm_port_t * const p_src_port, - IN const osm_port_t * const p_dest_port, - IN const ib_gid_t * const p_dgid, - IN const ib_net64_t comp_mask, - IN cl_qlist_t * const p_list) +pr_rcv_process_pair(IN osm_sa_t * sa, IN const osm_madw_t * const p_madw, + IN const osm_port_t * const requester_port, + IN const osm_port_t * const p_src_port, + IN const osm_port_t * const p_dest_port, + IN const ib_gid_t * const p_dgid, + IN const ib_net64_t comp_mask, + IN cl_qlist_t * const p_list) { OSM_LOG_ENTER(sa->p_log); - __osm_pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, - p_src_port, p_dest_port, p_dgid, - comp_mask, p_list); + pr_rcv_get_port_pair_paths(sa, p_madw, requester_port, p_src_port, + p_dest_port, p_dgid, comp_mask, p_list); OSM_LOG_EXIT(sa->p_log); } @@ -1507,9 +1484,8 @@ Exit: /********************************************************************** **********************************************************************/ static ib_api_status_t -__osm_pr_match_mgrp_attributes(IN osm_sa_t * sa, - IN const osm_madw_t * const p_madw, - IN const osm_mgrp_t * const p_mgrp) +pr_match_mgrp_attributes(IN osm_sa_t * sa, IN const osm_madw_t * const p_madw, + IN const osm_mgrp_t * const p_mgrp) { const ib_path_rec_t *p_pr; const ib_sa_mad_t *p_sa_mad; @@ -1573,8 +1549,7 @@ Exit: /********************************************************************** **********************************************************************/ static int -__osm_pr_rcv_check_mcast_dest(IN osm_sa_t * sa, - IN const osm_madw_t * const p_madw) +pr_rcv_check_mcast_dest(IN osm_sa_t * sa, IN const osm_madw_t * const p_madw) { const ib_path_rec_t *p_pr; const ib_sa_mad_t *p_sa_mad; @@ -1667,7 +1642,7 @@ void osm_pr_rcv_process(IN void *context, IN void *data) cl_plock_acquire(sa->p_lock); /* Handle multicast destinations separately */ - if ((ret = __osm_pr_rcv_check_mcast_dest(sa, p_madw)) < 0) { + if ((ret = pr_rcv_check_mcast_dest(sa, p_madw)) < 0) { /* Multicast DGID with unicast DLID */ cl_plock_release(sa->p_lock); osm_sa_send_error(sa, p_madw, IB_MAD_STATUS_INVALID_FIELD); @@ -1679,9 +1654,8 @@ void osm_pr_rcv_process(IN void *context, IN void *data) OSM_LOG(sa->p_log, OSM_LOG_DEBUG, "Unicast destination requested\n"); - sa_status = __osm_pr_rcv_get_end_points(sa, p_madw, - &p_src_port, &p_dest_port, - &dgid); + sa_status = pr_rcv_get_end_points(sa, p_madw, &p_src_port, &p_dest_port, + &dgid); if (sa_status == IB_SA_MAD_STATUS_SUCCESS) { /* @@ -1690,35 +1664,28 @@ void osm_pr_rcv_process(IN void *context, IN void *data) */ if (p_src_port) { if (p_dest_port) - __osm_pr_rcv_process_pair(sa, p_madw, - requester_port, - p_src_port, - p_dest_port, &dgid, - p_sa_mad->comp_mask, - &pr_list); + pr_rcv_process_pair(sa, p_madw, requester_port, + p_src_port, p_dest_port, + &dgid, p_sa_mad->comp_mask, + &pr_list); else - __osm_pr_rcv_process_half(sa, p_madw, - requester_port, - p_src_port, NULL, - &dgid, - p_sa_mad->comp_mask, - &pr_list); + pr_rcv_process_half(sa, p_madw, requester_port, + p_src_port, NULL, &dgid, + p_sa_mad->comp_mask, + &pr_list); } else { if (p_dest_port) - __osm_pr_rcv_process_half(sa, p_madw, - requester_port, NULL, - p_dest_port, &dgid, - p_sa_mad->comp_mask, - &pr_list); + pr_rcv_process_half(sa, p_madw, requester_port, + NULL, p_dest_port, &dgid, + p_sa_mad->comp_mask, + &pr_list); else /* Katie, bar the door! */ - __osm_pr_rcv_process_world(sa, p_madw, - requester_port, - &dgid, - p_sa_mad->comp_mask, - &pr_list); + pr_rcv_process_world(sa, p_madw, requester_port, + &dgid, p_sa_mad->comp_mask, + &pr_list); } } goto Unlock; @@ -1740,7 +1707,7 @@ McastDest: goto Unlock; /* Make sure the rest of the PathRecord matches the MC group attributes */ - status = __osm_pr_match_mgrp_attributes(sa, p_madw, p_mgrp); + status = pr_match_mgrp_attributes(sa, p_madw, p_mgrp); if (status != IB_SUCCESS) { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 1F19: " "MC group attributes don't match PathRecord request\n"); diff --git a/opensm/opensm/osm_sa_pkey_record.c b/opensm/opensm/osm_sa_pkey_record.c index 70c6047..b19329b 100644 --- a/opensm/opensm/osm_sa_pkey_record.c +++ b/opensm/opensm/osm_sa_pkey_record.c @@ -67,10 +67,8 @@ typedef struct osm_pkey_search_ctxt { /********************************************************************** **********************************************************************/ static void -__osm_sa_pkey_create(IN osm_sa_t * sa, - IN osm_physp_t * const p_physp, - IN osm_pkey_search_ctxt_t * const p_ctxt, - IN uint16_t block) +sa_pkey_create(IN osm_sa_t * sa, IN osm_physp_t * const p_physp, + IN osm_pkey_search_ctxt_t * const p_ctxt, IN uint16_t block) { osm_pkey_item_t *p_rec_item; uint16_t lid; @@ -114,9 +112,8 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sa_pkey_check_physp(IN osm_sa_t * sa, - IN osm_physp_t * const p_physp, - osm_pkey_search_ctxt_t * const p_ctxt) +sa_pkey_check_physp(IN osm_sa_t * sa, IN osm_physp_t * const p_physp, + osm_pkey_search_ctxt_t * const p_ctxt) { ib_net64_t comp_mask = p_ctxt->comp_mask; uint16_t block, num_blocks; @@ -125,13 +122,13 @@ __osm_sa_pkey_check_physp(IN osm_sa_t * sa, /* we got here with the phys port - all is left is to get the right block */ if (comp_mask & IB_PKEY_COMPMASK_BLOCK) { - __osm_sa_pkey_create(sa, p_physp, p_ctxt, p_ctxt->block_num); + sa_pkey_create(sa, p_physp, p_ctxt, p_ctxt->block_num); } else { num_blocks = osm_pkey_tbl_get_num_blocks(osm_physp_get_pkey_tbl (p_physp)); for (block = 0; block < num_blocks; block++) { - __osm_sa_pkey_create(sa, p_physp, p_ctxt, block); + sa_pkey_create(sa, p_physp, p_ctxt, block); } } @@ -141,9 +138,8 @@ __osm_sa_pkey_check_physp(IN osm_sa_t * sa, /********************************************************************** **********************************************************************/ static void -__osm_sa_pkey_by_comp_mask(IN osm_sa_t * sa, - IN const osm_port_t * const p_port, - osm_pkey_search_ctxt_t * const p_ctxt) +sa_pkey_by_comp_mask(IN osm_sa_t * sa, IN const osm_port_t * const p_port, + osm_pkey_search_ctxt_t * const p_ctxt) { const ib_pkey_table_record_t *p_rcvd_rec; ib_net64_t comp_mask; @@ -179,8 +175,7 @@ __osm_sa_pkey_by_comp_mask(IN osm_sa_t * sa, if (p_physp && (osm_physp_share_pkey (sa->p_log, p_req_physp, p_physp))) - __osm_sa_pkey_check_physp(sa, p_physp, - p_ctxt); + sa_pkey_check_physp(sa, p_physp, p_ctxt); } else { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 4603: " "Given Physical Port Number: 0x%X is out of range should be < 0x%X\n", @@ -202,7 +197,7 @@ __osm_sa_pkey_by_comp_mask(IN osm_sa_t * sa, (sa->p_log, p_req_physp, p_physp)) continue; - __osm_sa_pkey_check_physp(sa, p_physp, p_ctxt); + sa_pkey_check_physp(sa, p_physp, p_ctxt); } } Exit: @@ -212,14 +207,13 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sa_pkey_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, - IN void *context) +sa_pkey_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, IN void *context) { const osm_port_t *const p_port = (osm_port_t *) p_map_item; osm_pkey_search_ctxt_t *const p_ctxt = (osm_pkey_search_ctxt_t *) context; - __osm_sa_pkey_by_comp_mask(p_ctxt->sa, p_port, p_ctxt); + sa_pkey_by_comp_mask(p_ctxt->sa, p_port, p_ctxt); } /********************************************************************** @@ -325,11 +319,10 @@ void osm_pkey_rec_rcv_process(IN void *ctx, IN void *data) /* if we got a unique port - no need for a port search */ if (p_port) /* this does the loop on all the port phys ports */ - __osm_sa_pkey_by_comp_mask(sa, p_port, &context); + sa_pkey_by_comp_mask(sa, p_port, &context); else cl_qmap_apply_func(&sa->p_subn->port_guid_tbl, - __osm_sa_pkey_by_comp_mask_cb, - &context); + sa_pkey_by_comp_mask_cb, &context); } cl_plock_release(sa->p_lock); diff --git a/opensm/opensm/osm_sa_portinfo_record.c b/opensm/opensm/osm_sa_portinfo_record.c index 753c3db..79bfaf2 100644 --- a/opensm/opensm/osm_sa_portinfo_record.c +++ b/opensm/opensm/osm_sa_portinfo_record.c @@ -75,9 +75,8 @@ typedef struct osm_pir_search_ctxt { /********************************************************************** **********************************************************************/ static ib_api_status_t -__osm_pir_rcv_new_pir(IN osm_sa_t * sa, - IN const osm_physp_t * const p_physp, - IN cl_qlist_t * const p_list, IN ib_net16_t const lid) +pir_rcv_new_pir(IN osm_sa_t * sa, IN const osm_physp_t * const p_physp, + IN cl_qlist_t * const p_list, IN ib_net16_t const lid) { osm_pir_item_t *p_rec_item; ib_api_status_t status = IB_SUCCESS; @@ -114,9 +113,8 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sa_pir_create(IN osm_sa_t * sa, - IN const osm_physp_t * const p_physp, - IN osm_pir_search_ctxt_t * const p_ctxt) +sa_pir_create(IN osm_sa_t * sa, IN const osm_physp_t * const p_physp, + IN osm_pir_search_ctxt_t * const p_ctxt) { uint8_t lmc; uint16_t max_lid_ho; @@ -154,8 +152,7 @@ __osm_sa_pir_create(IN osm_sa_t * sa, goto Exit; } - __osm_pir_rcv_new_pir(sa, p_physp, p_ctxt->p_list, - cl_hton16(base_lid_ho)); + pir_rcv_new_pir(sa, p_physp, p_ctxt->p_list, cl_hton16(base_lid_ho)); Exit: OSM_LOG_EXIT(sa->p_log); @@ -164,9 +161,8 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sa_pir_check_physp(IN osm_sa_t * sa, - IN const osm_physp_t * const p_physp, - osm_pir_search_ctxt_t * const p_ctxt) +sa_pir_check_physp(IN osm_sa_t * sa, IN const osm_physp_t * const p_physp, + osm_pir_search_ctxt_t * const p_ctxt) { const ib_portinfo_record_t *p_rcvd_rec; ib_net64_t comp_mask; @@ -387,7 +383,7 @@ __osm_sa_pir_check_physp(IN osm_sa_t * sa, goto Exit; } - __osm_sa_pir_create(sa, p_physp, p_ctxt); + sa_pir_create(sa, p_physp, p_ctxt); Exit: OSM_LOG_EXIT(sa->p_log); @@ -396,9 +392,8 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sa_pir_by_comp_mask(IN osm_sa_t * sa, - IN osm_node_t * const p_node, - osm_pir_search_ctxt_t * const p_ctxt) +sa_pir_by_comp_mask(IN osm_sa_t * sa, IN osm_node_t * const p_node, + osm_pir_search_ctxt_t * const p_ctxt) { const ib_portinfo_record_t *p_rcvd_rec; ib_net64_t comp_mask; @@ -425,8 +420,7 @@ __osm_sa_pir_by_comp_mask(IN osm_sa_t * sa, if (p_physp && osm_physp_share_pkey(sa->p_log, p_req_physp, p_physp)) - __osm_sa_pir_check_physp(sa, p_physp, - p_ctxt); + sa_pir_check_physp(sa, p_physp, p_ctxt); } } else { for (port_num = 0; port_num < num_ports; port_num++) { @@ -441,7 +435,7 @@ __osm_sa_pir_by_comp_mask(IN osm_sa_t * sa, (sa->p_log, p_req_physp, p_physp)) continue; - __osm_sa_pir_check_physp(sa, p_physp, p_ctxt); + sa_pir_check_physp(sa, p_physp, p_ctxt); } } @@ -451,13 +445,12 @@ __osm_sa_pir_by_comp_mask(IN osm_sa_t * sa, /********************************************************************** **********************************************************************/ static void -__osm_sa_pir_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, - IN void *context) +sa_pir_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, IN void *context) { osm_node_t *const p_node = (osm_node_t *) p_map_item; osm_pir_search_ctxt_t *const p_ctxt = (osm_pir_search_ctxt_t *) context; - __osm_sa_pir_by_comp_mask(p_ctxt->sa, p_node, p_ctxt); + sa_pir_by_comp_mask(p_ctxt->sa, p_node, p_ctxt); } /********************************************************************** @@ -560,12 +553,10 @@ void osm_pir_rcv_process(IN void *ctx, IN void *data) if (status == IB_SUCCESS) { if (p_port) - __osm_sa_pir_by_comp_mask(sa, p_port->p_node, - &context); + sa_pir_by_comp_mask(sa, p_port->p_node, &context); else cl_qmap_apply_func(&sa->p_subn->node_guid_tbl, - __osm_sa_pir_by_comp_mask_cb, - &context); + sa_pir_by_comp_mask_cb, &context); } cl_plock_release(sa->p_lock); diff --git a/opensm/opensm/osm_sa_service_record.c b/opensm/opensm/osm_sa_service_record.c index 66b4cb7..0a6d681 100644 --- a/opensm/opensm/osm_sa_service_record.c +++ b/opensm/opensm/osm_sa_service_record.c @@ -205,9 +205,8 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sr_rcv_respond(IN osm_sa_t * sa, - IN osm_madw_t * const p_madw, - IN cl_qlist_t * const p_list) +sr_rcv_respond(IN osm_sa_t * sa, IN osm_madw_t * const p_madw, + IN cl_qlist_t * const p_list) { /* p923 - The ServiceKey shall be set to 0, except in the case of a trusted request. @@ -516,7 +515,7 @@ osm_sr_rcv_process_get_method(IN osm_sa_t * sa, goto Exit; } - __osm_sr_rcv_respond(sa, p_madw, &sr_match_item.sr_list); + sr_rcv_respond(sa, p_madw, &sr_match_item.sr_list); Exit: OSM_LOG_EXIT(sa->p_log); @@ -626,7 +625,7 @@ osm_sr_rcv_process_set_method(IN osm_sa_t * sa, cl_qlist_insert_tail(&sr_list, &p_sr_item->list_item); - __osm_sr_rcv_respond(sa, p_madw, &sr_list); + sr_rcv_respond(sa, p_madw, &sr_list); Exit: OSM_LOG_EXIT(sa->p_log); @@ -635,8 +634,7 @@ Exit: /********************************************************************** **********************************************************************/ static void -osm_sr_rcv_process_delete_method(IN osm_sa_t * sa, - IN osm_madw_t * const p_madw) +sr_rcv_process_delete_method(IN osm_sa_t * sa, IN osm_madw_t * const p_madw) { ib_sa_mad_t *p_sa_mad; ib_service_record_t *p_recvd_service_rec; @@ -695,7 +693,7 @@ osm_sr_rcv_process_delete_method(IN osm_sa_t * sa, if (p_svcr) osm_svcr_delete(p_svcr); - __osm_sr_rcv_respond(sa, p_madw, &sr_list); + sr_rcv_respond(sa, p_madw, &sr_list); Exit: OSM_LOG_EXIT(sa->p_log); @@ -740,7 +738,7 @@ void osm_sr_rcv_process(IN void *context, IN void *data) IB_SA_MAD_STATUS_REQ_INVALID); goto Exit; } - osm_sr_rcv_process_delete_method(sa, p_madw); + sr_rcv_process_delete_method(sa, p_madw); break; case IB_MAD_METHOD_GET: case IB_MAD_METHOD_GETTABLE: diff --git a/opensm/opensm/osm_sa_slvl_record.c b/opensm/opensm/osm_sa_slvl_record.c index 0550fc1..d65c60c 100644 --- a/opensm/opensm/osm_sa_slvl_record.c +++ b/opensm/opensm/osm_sa_slvl_record.c @@ -74,10 +74,8 @@ typedef struct osm_slvl_search_ctxt { /********************************************************************** **********************************************************************/ static void -__osm_sa_slvl_create(IN osm_sa_t * sa, - IN const osm_physp_t * const p_physp, - IN osm_slvl_search_ctxt_t * const p_ctxt, - IN uint8_t in_port_idx) +sa_slvl_create(IN osm_sa_t * sa, IN const osm_physp_t * const p_physp, + IN osm_slvl_search_ctxt_t * const p_ctxt, IN uint8_t in_port_idx) { osm_slvl_item_t *p_rec_item; uint16_t lid; @@ -121,9 +119,8 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sa_slvl_by_comp_mask(IN osm_sa_t * sa, - IN const osm_port_t * const p_port, - osm_slvl_search_ctxt_t * const p_ctxt) +sa_slvl_by_comp_mask(IN osm_sa_t * sa, IN const osm_port_t * const p_port, + osm_slvl_search_ctxt_t * const p_ctxt) { const ib_slvl_table_record_t *p_rcvd_rec; ib_net64_t comp_mask; @@ -153,7 +150,7 @@ __osm_sa_slvl_by_comp_mask(IN osm_sa_t * sa, /* check that the p_out_physp and the p_req_physp share a pkey */ if (osm_physp_share_pkey (sa->p_log, p_req_physp, p_out_physp)) - __osm_sa_slvl_create(sa, p_out_physp, p_ctxt, 0); + sa_slvl_create(sa, p_out_physp, p_ctxt, 0); } else { if (comp_mask & IB_SLVL_COMPMASK_OUT_PORT) out_port_start = out_port_end = @@ -188,8 +185,8 @@ __osm_sa_slvl_by_comp_mask(IN osm_sa_t * sa, (sa->p_log, p_req_physp, p_out_physp)) continue; - __osm_sa_slvl_create(sa, p_out_physp, p_ctxt, - in_port_num); + sa_slvl_create(sa, p_out_physp, p_ctxt, + in_port_num); } } } @@ -199,14 +196,13 @@ __osm_sa_slvl_by_comp_mask(IN osm_sa_t * sa, /********************************************************************** **********************************************************************/ static void -__osm_sa_slvl_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, - IN void *context) +sa_slvl_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, IN void *context) { const osm_port_t *const p_port = (osm_port_t *) p_map_item; osm_slvl_search_ctxt_t *const p_ctxt = (osm_slvl_search_ctxt_t *) context; - __osm_sa_slvl_by_comp_mask(p_ctxt->sa, p_port, p_ctxt); + sa_slvl_by_comp_mask(p_ctxt->sa, p_port, p_ctxt); } /********************************************************************** @@ -297,11 +293,10 @@ void osm_slvl_rec_rcv_process(IN void *ctx, IN void *data) /* if we have a unique port - no need for a port search */ if (p_port) /* this does the loop on all the port phys ports */ - __osm_sa_slvl_by_comp_mask(sa, p_port, &context); + sa_slvl_by_comp_mask(sa, p_port, &context); else cl_qmap_apply_func(&sa->p_subn->port_guid_tbl, - __osm_sa_slvl_by_comp_mask_cb, - &context); + sa_slvl_by_comp_mask_cb, &context); } cl_plock_release(sa->p_lock); diff --git a/opensm/opensm/osm_sa_sminfo_record.c b/opensm/opensm/osm_sa_sminfo_record.c index de99065..1898ae7 100644 --- a/opensm/opensm/osm_sa_sminfo_record.c +++ b/opensm/opensm/osm_sa_sminfo_record.c @@ -77,13 +77,10 @@ typedef struct osm_smir_search_ctxt { } osm_smir_search_ctxt_t; static ib_api_status_t -__osm_smir_rcv_new_smir(IN osm_sa_t * sa, - IN const osm_port_t * const p_port, - IN cl_qlist_t * const p_list, - IN ib_net64_t const guid, - IN ib_net32_t const act_count, - IN uint8_t const pri_state, - IN const osm_physp_t * const p_req_physp) +smir_rcv_new_smir(IN osm_sa_t * sa, IN const osm_port_t * const p_port, + IN cl_qlist_t * const p_list, IN ib_net64_t const guid, + IN ib_net32_t const act_count, IN uint8_t const pri_state, + IN const osm_physp_t * const p_req_physp) { osm_smir_item_t *p_rec_item; ib_api_status_t status = IB_SUCCESS; @@ -118,9 +115,9 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sa_smir_by_comp_mask(IN osm_sa_t * sa, - IN const osm_remote_sm_t * const p_rem_sm, - osm_smir_search_ctxt_t * const p_ctxt) +sa_smir_by_comp_mask(IN osm_sa_t * sa, + IN const osm_remote_sm_t * const p_rem_sm, + osm_smir_search_ctxt_t * const p_ctxt) { const ib_sminfo_record_t *const p_rcvd_rec = p_ctxt->p_rcvd_rec; const osm_physp_t *const p_req_physp = p_ctxt->p_req_physp; @@ -147,10 +144,9 @@ __osm_sa_smir_by_comp_mask(IN osm_sa_t * sa, /* Implement any other needed search cases */ - __osm_smir_rcv_new_smir(sa, p_rem_sm->p_port, p_ctxt->p_list, - p_rem_sm->smi.guid, - p_rem_sm->smi.act_count, - p_rem_sm->smi.pri_state, p_req_physp); + smir_rcv_new_smir(sa, p_rem_sm->p_port, p_ctxt->p_list, + p_rem_sm->smi.guid, p_rem_sm->smi.act_count, + p_rem_sm->smi.pri_state, p_req_physp); Exit: OSM_LOG_EXIT(sa->p_log); @@ -159,14 +155,13 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sa_smir_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, - IN void *context) +sa_smir_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, IN void *context) { const osm_remote_sm_t *const p_rem_sm = (osm_remote_sm_t *) p_map_item; osm_smir_search_ctxt_t *const p_ctxt = (osm_smir_search_ctxt_t *) context; - __osm_sa_smir_by_comp_mask(p_ctxt->sa, p_rem_sm, p_ctxt); + sa_smir_by_comp_mask(p_ctxt->sa, p_rem_sm, p_ctxt); } /********************************************************************** @@ -292,10 +287,10 @@ void osm_smir_rcv_process(IN void *ctx, IN void *data) pri_state = sa->p_subn->sm_state & 0x0F; pri_state |= (sa->p_subn->opt.sm_priority & 0x0F) << 4; - __osm_smir_rcv_new_smir(sa, local_port, context.p_list, - sa->p_subn->sm_port_guid, - cl_ntoh32(sa->p_subn->p_osm->stats.qp0_mads_sent), - pri_state, p_req_physp); + smir_rcv_new_smir(sa, local_port, context.p_list, + sa->p_subn->sm_port_guid, + cl_ntoh32(sa->p_subn->p_osm->stats.qp0_mads_sent), + pri_state, p_req_physp); } Remotes: @@ -308,8 +303,7 @@ void osm_smir_rcv_process(IN void *ctx, IN void *data) port_guid); if (p_rem_sm != (osm_remote_sm_t *) cl_qmap_end(p_sm_guid_tbl)) - __osm_sa_smir_by_comp_mask(sa, p_rem_sm, - &context); + sa_smir_by_comp_mask(sa, p_rem_sm, &context); else OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 280A: " "No remote SM for GUID 0x%016" PRIx64 @@ -317,8 +311,7 @@ void osm_smir_rcv_process(IN void *ctx, IN void *data) } else { /* Go over all other known (remote) SMs */ cl_qmap_apply_func(&sa->p_subn->sm_guid_tbl, - __osm_sa_smir_by_comp_mask_cb, - &context); + sa_smir_by_comp_mask_cb, &context); } } diff --git a/opensm/opensm/osm_sa_sw_info_record.c b/opensm/opensm/osm_sa_sw_info_record.c index 649b0ac..96ebb81 100644 --- a/opensm/opensm/osm_sa_sw_info_record.c +++ b/opensm/opensm/osm_sa_sw_info_record.c @@ -70,9 +70,8 @@ typedef struct osm_sir_search_ctxt { /********************************************************************** **********************************************************************/ static ib_api_status_t -__osm_sir_rcv_new_sir(IN osm_sa_t * sa, - IN const osm_switch_t * const p_sw, - IN cl_qlist_t * const p_list, IN ib_net16_t const lid) +sir_rcv_new_sir(IN osm_sa_t * sa, IN const osm_switch_t * const p_sw, + IN cl_qlist_t * const p_list, IN ib_net16_t const lid) { osm_sir_item_t *p_rec_item; ib_api_status_t status = IB_SUCCESS; @@ -105,11 +104,9 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sir_rcv_create_sir(IN osm_sa_t * sa, - IN const osm_switch_t * const p_sw, - IN cl_qlist_t * const p_list, - IN ib_net16_t const match_lid, - IN const osm_physp_t * const p_req_physp) +sir_rcv_create_sir(IN osm_sa_t * sa, IN const osm_switch_t * const p_sw, + IN cl_qlist_t * const p_list, IN ib_net16_t const match_lid, + IN const osm_physp_t * const p_req_physp) { osm_port_t *p_port; const osm_physp_t *p_physp; @@ -163,7 +160,7 @@ __osm_sir_rcv_create_sir(IN osm_sa_t * sa, } - __osm_sir_rcv_new_sir(sa, p_sw, p_list, osm_port_get_base_lid(p_port)); + sir_rcv_new_sir(sa, p_sw, p_list, osm_port_get_base_lid(p_port)); Exit: OSM_LOG_EXIT(sa->p_log); @@ -172,8 +169,7 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sir_rcv_by_comp_mask(IN cl_map_item_t * const p_map_item, - IN void *context) +sir_rcv_by_comp_mask(IN cl_map_item_t * const p_map_item, IN void *context) { const osm_sir_search_ctxt_t *const p_ctxt = (osm_sir_search_ctxt_t *) context; @@ -195,8 +191,7 @@ __osm_sir_rcv_by_comp_mask(IN cl_map_item_t * const p_map_item, goto Exit; } - __osm_sir_rcv_create_sir(sa, p_sw, p_ctxt->p_list, - match_lid, p_req_physp); + sir_rcv_create_sir(sa, p_sw, p_ctxt->p_list, match_lid, p_req_physp); Exit: OSM_LOG_EXIT(p_ctxt->sa->p_log); @@ -261,8 +256,8 @@ void osm_sir_rcv_process(IN void *ctx, IN void *data) cl_plock_acquire(sa->p_lock); /* Go over all switches */ - cl_qmap_apply_func(&sa->p_subn->sw_guid_tbl, - __osm_sir_rcv_by_comp_mask, &context); + cl_qmap_apply_func(&sa->p_subn->sw_guid_tbl, sir_rcv_by_comp_mask, + &context); cl_plock_release(sa->p_lock); diff --git a/opensm/opensm/osm_sa_vlarb_record.c b/opensm/opensm/osm_sa_vlarb_record.c index 3bc9b8a..7d20786 100644 --- a/opensm/opensm/osm_sa_vlarb_record.c +++ b/opensm/opensm/osm_sa_vlarb_record.c @@ -74,10 +74,9 @@ typedef struct osm_vl_arb_search_ctxt { /********************************************************************** **********************************************************************/ static void -__osm_sa_vl_arb_create(IN osm_sa_t * sa, - IN osm_physp_t * const p_physp, - IN osm_vl_arb_search_ctxt_t * const p_ctxt, - IN uint8_t block) +sa_vl_arb_create(IN osm_sa_t * sa, IN osm_physp_t * const p_physp, + IN osm_vl_arb_search_ctxt_t * const p_ctxt, + IN uint8_t block) { osm_vl_arb_item_t *p_rec_item; uint16_t lid; @@ -120,9 +119,8 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sa_vl_arb_check_physp(IN osm_sa_t * sa, - IN osm_physp_t * const p_physp, - osm_vl_arb_search_ctxt_t * const p_ctxt) +sa_vl_arb_check_physp(IN osm_sa_t * sa, IN osm_physp_t * const p_physp, + osm_vl_arb_search_ctxt_t * const p_ctxt) { ib_net64_t comp_mask = p_ctxt->comp_mask; uint8_t block; @@ -133,7 +131,7 @@ __osm_sa_vl_arb_check_physp(IN osm_sa_t * sa, for (block = 1; block <= 4; block++) { if (!(comp_mask & IB_VLA_COMPMASK_BLOCK) || block == p_ctxt->block_num) { - __osm_sa_vl_arb_create(sa, p_physp, p_ctxt, block); + sa_vl_arb_create(sa, p_physp, p_ctxt, block); } } @@ -143,9 +141,8 @@ __osm_sa_vl_arb_check_physp(IN osm_sa_t * sa, /********************************************************************** **********************************************************************/ static void -__osm_sa_vl_arb_by_comp_mask(IN osm_sa_t * sa, - IN const osm_port_t * const p_port, - osm_vl_arb_search_ctxt_t * const p_ctxt) +sa_vl_arb_by_comp_mask(IN osm_sa_t * sa, IN const osm_port_t * const p_port, + osm_vl_arb_search_ctxt_t * const p_ctxt) { const ib_vl_arb_table_record_t *p_rcvd_rec; ib_net64_t comp_mask; @@ -181,8 +178,7 @@ __osm_sa_vl_arb_by_comp_mask(IN osm_sa_t * sa, if (p_physp && osm_physp_share_pkey(sa->p_log, p_req_physp, p_physp)) - __osm_sa_vl_arb_check_physp(sa, p_physp, - p_ctxt); + sa_vl_arb_check_physp(sa, p_physp, p_ctxt); } else { OSM_LOG(sa->p_log, OSM_LOG_ERROR, "ERR 2A03: " "Given Physical Port Number: 0x%X is out of range should be < 0x%X\n", @@ -204,7 +200,7 @@ __osm_sa_vl_arb_by_comp_mask(IN osm_sa_t * sa, (sa->p_log, p_req_physp, p_physp)) continue; - __osm_sa_vl_arb_check_physp(sa, p_physp, p_ctxt); + sa_vl_arb_check_physp(sa, p_physp, p_ctxt); } } Exit: @@ -214,14 +210,14 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sa_vl_arb_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, - IN void *context) +sa_vl_arb_by_comp_mask_cb(IN cl_map_item_t * const p_map_item, + IN void *context) { const osm_port_t *const p_port = (osm_port_t *) p_map_item; osm_vl_arb_search_ctxt_t *const p_ctxt = (osm_vl_arb_search_ctxt_t *) context; - __osm_sa_vl_arb_by_comp_mask(p_ctxt->sa, p_port, p_ctxt); + sa_vl_arb_by_comp_mask(p_ctxt->sa, p_port, p_ctxt); } /********************************************************************** @@ -314,10 +310,10 @@ void osm_vlarb_rec_rcv_process(IN void *ctx, IN void *data) /* if we got a unique port - no need for a port search */ if (p_port) /* this does the loop on all the port phys ports */ - __osm_sa_vl_arb_by_comp_mask(sa, p_port, &context); + sa_vl_arb_by_comp_mask(sa, p_port, &context); else cl_qmap_apply_func(&sa->p_subn->port_guid_tbl, - __osm_sa_vl_arb_by_comp_mask_cb, + sa_vl_arb_by_comp_mask_cb, &context); } diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c index 60dcdbb..f17119b 100644 --- a/opensm/opensm/osm_sm.c +++ b/opensm/opensm/osm_sm.c @@ -90,7 +90,7 @@ static void osm_sm_process(osm_sm_t * sm, osm_signal_t signal) osm_state_mgr_process(sm, signal); } -static void __osm_sm_sweeper(IN void *p_ptr) +static void sm_sweeper(IN void *p_ptr) { ib_api_status_t status; osm_sm_t *const p_sm = (osm_sm_t *) p_ptr; @@ -390,7 +390,7 @@ osm_sm_init(IN osm_sm_t * const p_sm, * the sweeper thread if the user wants sweeping. */ p_sm->thread_state = OSM_THREAD_STATE_RUN; - status = cl_thread_init(&p_sm->sweeper, __osm_sm_sweeper, p_sm, + status = cl_thread_init(&p_sm->sweeper, sm_sweeper, p_sm, "opensm sweeper"); if (status != IB_SUCCESS) goto Exit; @@ -449,8 +449,7 @@ Exit: /********************************************************************** **********************************************************************/ static ib_api_status_t -__osm_sm_mgrp_process(IN osm_sm_t * const p_sm, - IN osm_mgrp_t * const p_mgrp) +sm_mgrp_process(IN osm_sm_t * const p_sm, IN osm_mgrp_t * const p_mgrp) { osm_mcast_mgr_ctxt_t *ctx; @@ -553,7 +552,7 @@ osm_sm_mcgrp_join(IN osm_sm_t * const p_sm, goto Exit; } - status = __osm_sm_mgrp_process(p_sm, p_mgrp); + status = sm_mgrp_process(p_sm, p_mgrp); CL_PLOCK_RELEASE(p_sm->p_lock); Exit: @@ -609,7 +608,7 @@ osm_sm_mcgrp_leave(IN osm_sm_t * const p_sm, */ osm_port_remove_mgrp(p_port, mlid); - status = __osm_sm_mgrp_process(p_sm, p_mgrp); + status = sm_mgrp_process(p_sm, p_mgrp); CL_PLOCK_RELEASE(p_sm->p_lock); Exit: diff --git a/opensm/opensm/osm_sm_mad_ctrl.c b/opensm/opensm/osm_sm_mad_ctrl.c index 992d712..04b23d7 100644 --- a/opensm/opensm/osm_sm_mad_ctrl.c +++ b/opensm/opensm/osm_sm_mad_ctrl.c @@ -55,9 +55,9 @@ #include #include -/****f* opensm: SM/__osm_sm_mad_ctrl_retire_trans_mad +/****f* opensm: SM/sm_mad_ctrl_retire_trans_mad * NAME - * __osm_sm_mad_ctrl_retire_trans_mad + * sm_mad_ctrl_retire_trans_mad * * DESCRIPTION * This function handles clean-up of MADs associated with the SM's @@ -67,7 +67,7 @@ */ static void -__osm_sm_mad_ctrl_retire_trans_mad(IN osm_sm_mad_ctrl_t * const p_ctrl, +sm_mad_ctrl_retire_trans_mad(IN osm_sm_mad_ctrl_t * const p_ctrl, IN osm_madw_t * const p_madw) { uint32_t outstanding; @@ -95,9 +95,9 @@ __osm_sm_mad_ctrl_retire_trans_mad(IN osm_sm_mad_ctrl_t * const p_ctrl, /************/ -/****f* opensm: SM/__osm_sm_mad_ctrl_disp_done_callback +/****f* opensm: SM/sm_mad_ctrl_disp_done_callback * NAME - * __osm_sm_mad_ctrl_disp_done_callback + * sm_mad_ctrl_disp_done_callback * * DESCRIPTION * This function is the Dispatcher callback that indicates @@ -106,7 +106,7 @@ __osm_sm_mad_ctrl_retire_trans_mad(IN osm_sm_mad_ctrl_t * const p_ctrl, * SYNOPSIS */ static void -__osm_sm_mad_ctrl_disp_done_callback(IN void *context, IN void *p_data) +sm_mad_ctrl_disp_done_callback(IN void *context, IN void *p_data) { osm_sm_mad_ctrl_t *const p_ctrl = (osm_sm_mad_ctrl_t *) context; osm_madw_t *const p_madw = (osm_madw_t *) p_data; @@ -127,9 +127,9 @@ __osm_sm_mad_ctrl_disp_done_callback(IN void *context, IN void *p_data) p_smp = osm_madw_get_smp_ptr(p_madw); if (ib_smp_is_response(p_smp)) { CL_ASSERT(p_madw->resp_expected == FALSE); - __osm_sm_mad_ctrl_retire_trans_mad(p_ctrl, p_madw); + sm_mad_ctrl_retire_trans_mad(p_ctrl, p_madw); } else if (p_madw->resp_expected == TRUE) - __osm_sm_mad_ctrl_retire_trans_mad(p_ctrl, p_madw); + sm_mad_ctrl_retire_trans_mad(p_ctrl, p_madw); else osm_mad_pool_put(p_ctrl->p_mad_pool, p_madw); @@ -138,9 +138,9 @@ __osm_sm_mad_ctrl_disp_done_callback(IN void *context, IN void *p_data) /************/ -/****f* opensm: SM/__osm_sm_mad_ctrl_update_wire_stats +/****f* opensm: SM/sm_mad_ctrl_update_wire_stats * NAME - * __osm_sm_mad_ctrl_update_wire_stats + * sm_mad_ctrl_update_wire_stats * * DESCRIPTION * Updates wire stats for outstanding MADs and calls the VL15 poller. @@ -148,7 +148,7 @@ __osm_sm_mad_ctrl_disp_done_callback(IN void *context, IN void *p_data) * SYNOPSIS */ static void -__osm_sm_mad_ctrl_update_wire_stats(IN osm_sm_mad_ctrl_t * const p_ctrl) +sm_mad_ctrl_update_wire_stats(IN osm_sm_mad_ctrl_t * const p_ctrl) { uint32_t mads_on_wire; @@ -169,9 +169,9 @@ __osm_sm_mad_ctrl_update_wire_stats(IN osm_sm_mad_ctrl_t * const p_ctrl) OSM_LOG_EXIT(p_ctrl->p_log); } -/****f* opensm: SM/__osm_sm_mad_ctrl_process_get_resp +/****f* opensm: SM/sm_mad_ctrl_process_get_resp * NAME - * __osm_sm_mad_ctrl_process_get_resp + * sm_mad_ctrl_process_get_resp * * DESCRIPTION * This function handles method GetResp() for received MADs. @@ -180,7 +180,7 @@ __osm_sm_mad_ctrl_update_wire_stats(IN osm_sm_mad_ctrl_t * const p_ctrl) * SYNOPSIS */ static void -__osm_sm_mad_ctrl_process_get_resp(IN osm_sm_mad_ctrl_t * const p_ctrl, +sm_mad_ctrl_process_get_resp(IN osm_sm_mad_ctrl_t * const p_ctrl, IN osm_madw_t * p_madw, IN void *transaction_context) { @@ -204,7 +204,7 @@ __osm_sm_mad_ctrl_process_get_resp(IN osm_sm_mad_ctrl_t * const p_ctrl, p_old_madw = (osm_madw_t *) transaction_context; - __osm_sm_mad_ctrl_update_wire_stats(p_ctrl); + sm_mad_ctrl_update_wire_stats(p_ctrl); /* Copy the MAD Wrapper context from the requesting MAD @@ -277,7 +277,7 @@ __osm_sm_mad_ctrl_process_get_resp(IN osm_sm_mad_ctrl_t * const p_ctrl, osm_get_disp_msg_str(msg_id)); status = cl_disp_post(p_ctrl->h_disp, msg_id, p_madw, - __osm_sm_mad_ctrl_disp_done_callback, p_ctrl); + sm_mad_ctrl_disp_done_callback, p_ctrl); if (status != CL_SUCCESS) { OSM_LOG(p_ctrl->p_log, OSM_LOG_ERROR, "ERR 3104: " @@ -290,9 +290,9 @@ Exit: OSM_LOG_EXIT(p_ctrl->p_log); } -/****f* opensm: SM/__osm_sm_mad_ctrl_process_get +/****f* opensm: SM/sm_mad_ctrl_process_get * NAME - * __osm_sm_mad_ctrl_process_get + * sm_mad_ctrl_process_get * * DESCRIPTION * This function handles method Get() for received MADs. @@ -300,8 +300,8 @@ Exit: * SYNOPSIS */ static void -__osm_sm_mad_ctrl_process_get(IN osm_sm_mad_ctrl_t * const p_ctrl, - IN osm_madw_t * p_madw) +sm_mad_ctrl_process_get(IN osm_sm_mad_ctrl_t * const p_ctrl, + IN osm_madw_t * p_madw) { ib_smp_t *p_smp; cl_status_t status; @@ -346,7 +346,7 @@ __osm_sm_mad_ctrl_process_get(IN osm_sm_mad_ctrl_t * const p_ctrl, osm_get_disp_msg_str(msg_id)); status = cl_disp_post(p_ctrl->h_disp, msg_id, p_madw, - __osm_sm_mad_ctrl_disp_done_callback, p_ctrl); + sm_mad_ctrl_disp_done_callback, p_ctrl); if (status != CL_SUCCESS) { OSM_LOG(p_ctrl->p_log, OSM_LOG_ERROR, "ERR 3106: " @@ -369,9 +369,9 @@ Exit: * SEE ALSO *********/ -/****f* opensm: SM/__osm_sm_mad_ctrl_process_set +/****f* opensm: SM/sm_mad_ctrl_process_set * NAME - * __osm_sm_mad_ctrl_process_set + * sm_mad_ctrl_process_set * * DESCRIPTION * This function handles method Set() for received MADs. @@ -379,8 +379,8 @@ Exit: * SYNOPSIS */ static void -__osm_sm_mad_ctrl_process_set(IN osm_sm_mad_ctrl_t * const p_ctrl, - IN osm_madw_t * p_madw) +sm_mad_ctrl_process_set(IN osm_sm_mad_ctrl_t * const p_ctrl, + IN osm_madw_t * p_madw) { ib_smp_t *p_smp; cl_status_t status; @@ -426,7 +426,7 @@ __osm_sm_mad_ctrl_process_set(IN osm_sm_mad_ctrl_t * const p_ctrl, osm_get_disp_msg_str(msg_id)); status = cl_disp_post(p_ctrl->h_disp, msg_id, p_madw, - __osm_sm_mad_ctrl_disp_done_callback, p_ctrl); + sm_mad_ctrl_disp_done_callback, p_ctrl); if (status != CL_SUCCESS) { OSM_LOG(p_ctrl->p_log, OSM_LOG_ERROR, "ERR 3108: " @@ -449,9 +449,9 @@ Exit: * SEE ALSO *********/ -/****f* opensm: SM/__osm_sm_mad_ctrl_process_trap +/****f* opensm: SM/sm_mad_ctrl_process_trap * NAME - * __osm_sm_mad_ctrl_process_trap + * sm_mad_ctrl_process_trap * * DESCRIPTION * This function handles method Trap() for received MADs. @@ -459,8 +459,8 @@ Exit: * SYNOPSIS */ static void -__osm_sm_mad_ctrl_process_trap(IN osm_sm_mad_ctrl_t * const p_ctrl, - IN osm_madw_t * p_madw) +sm_mad_ctrl_process_trap(IN osm_sm_mad_ctrl_t * const p_ctrl, + IN osm_madw_t * p_madw) { ib_smp_t *p_smp; cl_status_t status; @@ -515,7 +515,7 @@ __osm_sm_mad_ctrl_process_trap(IN osm_sm_mad_ctrl_t * const p_ctrl, osm_get_disp_msg_str(msg_id)); status = cl_disp_post(p_ctrl->h_disp, msg_id, p_madw, - __osm_sm_mad_ctrl_disp_done_callback, p_ctrl); + sm_mad_ctrl_disp_done_callback, p_ctrl); if (status != CL_SUCCESS) { OSM_LOG(p_ctrl->p_log, OSM_LOG_ERROR, "ERR 3110: " @@ -538,9 +538,9 @@ Exit: * SEE ALSO *********/ -/****f* opensm: SM/__osm_sm_mad_ctrl_process_trap_repress +/****f* opensm: SM/sm_mad_ctrl_process_trap_repress * NAME - * __osm_sm_mad_ctrl_process_trap_repress + * sm_mad_ctrl_process_trap_repress * * DESCRIPTION * This function handles method TrapRepress() for received MADs. @@ -548,8 +548,8 @@ Exit: * SYNOPSIS */ static void -__osm_sm_mad_ctrl_process_trap_repress(IN osm_sm_mad_ctrl_t * const p_ctrl, - IN osm_madw_t * p_madw) +sm_mad_ctrl_process_trap_repress(IN osm_sm_mad_ctrl_t * const p_ctrl, + IN osm_madw_t * p_madw) { ib_smp_t *p_smp; @@ -589,9 +589,9 @@ __osm_sm_mad_ctrl_process_trap_repress(IN osm_sm_mad_ctrl_t * const p_ctrl, * SEE ALSO *********/ -/****f* opensm: SM/__osm_sm_mad_ctrl_rcv_callback +/****f* opensm: SM/sm_mad_ctrl_rcv_callback * NAME - * __osm_sm_mad_ctrl_rcv_callback + * sm_mad_ctrl_rcv_callback * * DESCRIPTION * This is the callback from the transport layer for received MADs. @@ -599,9 +599,8 @@ __osm_sm_mad_ctrl_process_trap_repress(IN osm_sm_mad_ctrl_t * const p_ctrl, * SYNOPSIS */ static void -__osm_sm_mad_ctrl_rcv_callback(IN osm_madw_t * p_madw, - IN void *bind_context, - IN osm_madw_t * p_req_madw) +sm_mad_ctrl_rcv_callback(IN osm_madw_t * p_madw, IN void *bind_context, + IN osm_madw_t * p_req_madw) { osm_sm_mad_ctrl_t *p_ctrl = (osm_sm_mad_ctrl_t *) bind_context; ib_smp_t *p_smp; @@ -631,9 +630,9 @@ __osm_sm_mad_ctrl_rcv_callback(IN osm_madw_t * p_madw, /* retire the mad or put it back */ if (ib_smp_is_response(p_smp)) { CL_ASSERT(p_madw->resp_expected == FALSE); - __osm_sm_mad_ctrl_retire_trans_mad(p_ctrl, p_madw); + sm_mad_ctrl_retire_trans_mad(p_ctrl, p_madw); } else if (p_madw->resp_expected == TRUE) - __osm_sm_mad_ctrl_retire_trans_mad(p_ctrl, p_madw); + sm_mad_ctrl_retire_trans_mad(p_ctrl, p_madw); else osm_mad_pool_put(p_ctrl->p_mad_pool, p_madw); @@ -657,27 +656,27 @@ __osm_sm_mad_ctrl_rcv_callback(IN osm_madw_t * p_madw, switch (p_smp->method) { case IB_MAD_METHOD_GET_RESP: CL_ASSERT(p_req_madw != NULL); - __osm_sm_mad_ctrl_process_get_resp(p_ctrl, p_madw, p_req_madw); + sm_mad_ctrl_process_get_resp(p_ctrl, p_madw, p_req_madw); break; case IB_MAD_METHOD_GET: CL_ASSERT(p_req_madw == NULL); - __osm_sm_mad_ctrl_process_get(p_ctrl, p_madw); + sm_mad_ctrl_process_get(p_ctrl, p_madw); break; case IB_MAD_METHOD_TRAP: CL_ASSERT(p_req_madw == NULL); - __osm_sm_mad_ctrl_process_trap(p_ctrl, p_madw); + sm_mad_ctrl_process_trap(p_ctrl, p_madw); break; case IB_MAD_METHOD_SET: CL_ASSERT(p_req_madw == NULL); - __osm_sm_mad_ctrl_process_set(p_ctrl, p_madw); + sm_mad_ctrl_process_set(p_ctrl, p_madw); break; case IB_MAD_METHOD_TRAP_REPRESS: CL_ASSERT(p_req_madw != NULL); - __osm_sm_mad_ctrl_process_trap_repress(p_ctrl, p_madw); + sm_mad_ctrl_process_trap_repress(p_ctrl, p_madw); break; case IB_MAD_METHOD_SEND: @@ -706,9 +705,9 @@ Exit: * SEE ALSO *********/ -/****f* opensm: SM/__osm_sm_mad_ctrl_send_err_cb +/****f* opensm: SM/sm_mad_ctrl_send_err_cb * NAME - * __osm_sm_mad_ctrl_send_err_cb + * sm_mad_ctrl_send_err_cb * * DESCRIPTION * This is the callback from the transport layer for send errors @@ -717,7 +716,7 @@ Exit: * SYNOPSIS */ static void -__osm_sm_mad_ctrl_send_err_cb(IN void *bind_context, IN osm_madw_t * p_madw) +sm_mad_ctrl_send_err_cb(IN void *bind_context, IN osm_madw_t * p_madw) { osm_sm_mad_ctrl_t *p_ctrl = (osm_sm_mad_ctrl_t *) bind_context; ib_api_status_t status; @@ -782,7 +781,7 @@ __osm_sm_mad_ctrl_send_err_cb(IN void *bind_context, IN osm_madw_t * p_madw) osm_dump_dr_smp(p_ctrl->p_log, osm_madw_get_smp_ptr(p_madw), OSM_LOG_ERROR); - __osm_sm_mad_ctrl_update_wire_stats(p_ctrl); + sm_mad_ctrl_update_wire_stats(p_ctrl); if (osm_madw_get_err_msg(p_madw) != CL_DISP_MSGID_NONE) { OSM_LOG(p_ctrl->p_log, OSM_LOG_DEBUG, @@ -790,10 +789,8 @@ __osm_sm_mad_ctrl_send_err_cb(IN void *bind_context, IN osm_madw_t * p_madw) osm_get_disp_msg_str(osm_madw_get_err_msg(p_madw))); status = cl_disp_post(p_ctrl->h_disp, - osm_madw_get_err_msg(p_madw), - p_madw, - __osm_sm_mad_ctrl_disp_done_callback, - p_ctrl); + osm_madw_get_err_msg(p_madw), p_madw, + sm_mad_ctrl_disp_done_callback, p_ctrl); if (status != CL_SUCCESS) OSM_LOG(p_ctrl->p_log, OSM_LOG_ERROR, "ERR 3115: " "Dispatcher post message failed (%s)\n", @@ -802,7 +799,7 @@ __osm_sm_mad_ctrl_send_err_cb(IN void *bind_context, IN osm_madw_t * p_madw) /* No error message was provided, just retire the MAD. */ - __osm_sm_mad_ctrl_retire_trans_mad(p_ctrl, p_madw); + sm_mad_ctrl_retire_trans_mad(p_ctrl, p_madw); OSM_LOG_EXIT(p_ctrl->p_log); } @@ -910,11 +907,10 @@ osm_sm_mad_ctrl_bind(IN osm_sm_mad_ctrl_t * const p_ctrl, OSM_LOG(p_ctrl->p_log, OSM_LOG_VERBOSE, "Binding to port 0x%" PRIx64 "\n", cl_ntoh64(port_guid)); - p_ctrl->h_bind = osm_vendor_bind(p_ctrl->p_vendor, - &bind_info, + p_ctrl->h_bind = osm_vendor_bind(p_ctrl->p_vendor, &bind_info, p_ctrl->p_mad_pool, - __osm_sm_mad_ctrl_rcv_callback, - __osm_sm_mad_ctrl_send_err_cb, p_ctrl); + sm_mad_ctrl_rcv_callback, + sm_mad_ctrl_send_err_cb, p_ctrl); if (p_ctrl->h_bind == OSM_BIND_INVALID_HANDLE) { status = IB_ERROR; diff --git a/opensm/opensm/osm_sm_state_mgr.c b/opensm/opensm/osm_sm_state_mgr.c index 343a9e3..87af1ea 100644 --- a/opensm/opensm/osm_sm_state_mgr.c +++ b/opensm/opensm/osm_sm_state_mgr.c @@ -75,7 +75,7 @@ void osm_report_sm_state(osm_sm_t * sm) /********************************************************************** **********************************************************************/ -static void __osm_sm_state_mgr_send_master_sm_info_req(osm_sm_t * sm) +static void sm_state_mgr_send_master_sm_info_req(osm_sm_t * sm) { osm_madw_context_t context; const osm_port_t *p_port; @@ -124,7 +124,7 @@ Exit: /********************************************************************** **********************************************************************/ -static void __osm_sm_state_mgr_start_polling(osm_sm_t * sm) +static void sm_state_mgr_start_polling(osm_sm_t * sm) { uint32_t timeout = sm->p_subn->opt.sminfo_polling_timeout; cl_status_t cl_status; @@ -139,7 +139,7 @@ static void __osm_sm_state_mgr_start_polling(osm_sm_t * sm) /* * Send a SubnGet(SMInfo) query to the current (or new) master found. */ - __osm_sm_state_mgr_send_master_sm_info_req(sm); + sm_state_mgr_send_master_sm_info_req(sm); /* * Start a timer that will wake up every sminfo_polling_timeout milliseconds. @@ -207,7 +207,7 @@ void osm_sm_state_mgr_polling_callback(IN void *context) } /* Send a SubnGet(SMInfo) request to the remote sm (depends on our state) */ - __osm_sm_state_mgr_send_master_sm_info_req(sm); + sm_state_mgr_send_master_sm_info_req(sm); /* restart the timer */ cl_status = cl_timer_start(&sm->polling_timer, timeout); @@ -222,8 +222,8 @@ Exit: /********************************************************************** **********************************************************************/ -static void __osm_sm_state_mgr_signal_error(osm_sm_t * sm, - IN const osm_sm_signal_t signal) +static void sm_state_mgr_signal_error(osm_sm_t * sm, + IN const osm_sm_signal_t signal) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 3207: " "Invalid signal %s in state %s\n", @@ -293,7 +293,7 @@ ib_api_status_t osm_sm_state_mgr_process(osm_sm_t * sm, */ sm->p_subn->ignore_existing_lfts = FALSE; - __osm_sm_state_mgr_start_polling(sm); + sm_state_mgr_start_polling(sm); break; case OSM_SM_SIGNAL_HANDOVER: /* @@ -304,7 +304,7 @@ ib_api_status_t osm_sm_state_mgr_process(osm_sm_t * sm, */ break; default: - __osm_sm_state_mgr_signal_error(sm, signal); + sm_state_mgr_signal_error(sm, signal); status = IB_INVALID_PARAMETER; break; } @@ -361,7 +361,7 @@ ib_api_status_t osm_sm_state_mgr_process(osm_sm_t * sm, */ break; default: - __osm_sm_state_mgr_signal_error(sm, signal); + sm_state_mgr_signal_error(sm, signal); status = IB_INVALID_PARAMETER; break; } @@ -376,10 +376,10 @@ ib_api_status_t osm_sm_state_mgr_process(osm_sm_t * sm, */ sm->p_subn->sm_state = IB_SMINFO_STATE_STANDBY; osm_report_sm_state(sm); - __osm_sm_state_mgr_start_polling(sm); + sm_state_mgr_start_polling(sm); break; default: - __osm_sm_state_mgr_signal_error(sm, signal); + sm_state_mgr_signal_error(sm, signal); status = IB_INVALID_PARAMETER; break; } @@ -422,7 +422,7 @@ ib_api_status_t osm_sm_state_mgr_process(osm_sm_t * sm, */ sm->p_subn->sm_state = IB_SMINFO_STATE_STANDBY; osm_report_sm_state(sm); - __osm_sm_state_mgr_start_polling(sm); + sm_state_mgr_start_polling(sm); break; case OSM_SM_SIGNAL_WAIT_FOR_HANDOVER: /* @@ -432,14 +432,14 @@ ib_api_status_t osm_sm_state_mgr_process(osm_sm_t * sm, * we should move back to discovering, since something must * have happened to it. */ - __osm_sm_state_mgr_start_polling(sm); + sm_state_mgr_start_polling(sm); break; case OSM_SM_SIGNAL_DISCOVER: sm->p_subn->sm_state = IB_SMINFO_STATE_DISCOVERING; osm_report_sm_state(sm); break; default: - __osm_sm_state_mgr_signal_error(sm, signal); + sm_state_mgr_signal_error(sm, signal); status = IB_INVALID_PARAMETER; break; } @@ -489,7 +489,7 @@ ib_api_status_t osm_sm_state_mgr_check_legality(osm_sm_t * sm, status = IB_SUCCESS; break; default: - __osm_sm_state_mgr_signal_error(sm, signal); + sm_state_mgr_signal_error(sm, signal); status = IB_INVALID_PARAMETER; break; } @@ -505,7 +505,7 @@ ib_api_status_t osm_sm_state_mgr_check_legality(osm_sm_t * sm, status = IB_SUCCESS; break; default: - __osm_sm_state_mgr_signal_error(sm, signal); + sm_state_mgr_signal_error(sm, signal); status = IB_INVALID_PARAMETER; break; } @@ -517,7 +517,7 @@ ib_api_status_t osm_sm_state_mgr_check_legality(osm_sm_t * sm, status = IB_SUCCESS; break; default: - __osm_sm_state_mgr_signal_error(sm, signal); + sm_state_mgr_signal_error(sm, signal); status = IB_INVALID_PARAMETER; break; } @@ -530,7 +530,7 @@ ib_api_status_t osm_sm_state_mgr_check_legality(osm_sm_t * sm, status = IB_SUCCESS; break; default: - __osm_sm_state_mgr_signal_error(sm, signal); + sm_state_mgr_signal_error(sm, signal); status = IB_INVALID_PARAMETER; break; } diff --git a/opensm/opensm/osm_sminfo_rcv.c b/opensm/opensm/osm_sminfo_rcv.c index 429a891..d3062e0 100644 --- a/opensm/opensm/osm_sminfo_rcv.c +++ b/opensm/opensm/osm_sminfo_rcv.c @@ -65,8 +65,8 @@ and lower GUID. **********************************************************************/ static inline boolean_t -__osm_sminfo_rcv_remote_sm_is_higher(IN osm_sm_t * sm, - IN const ib_sm_info_t * p_remote_smi) +sminfo_rcv_remote_sm_is_higher(IN osm_sm_t * sm, + IN const ib_sm_info_t * p_remote_smi) { return (osm_sm_is_greater_than(ib_sminfo_get_priority(p_remote_smi), p_remote_smi->guid, @@ -78,8 +78,8 @@ __osm_sminfo_rcv_remote_sm_is_higher(IN osm_sm_t * sm, /********************************************************************** **********************************************************************/ static void -__osm_sminfo_rcv_process_get_request(IN osm_sm_t * sm, - IN const osm_madw_t * const p_madw) +sminfo_rcv_process_get_request(IN osm_sm_t * sm, + IN const osm_madw_t * const p_madw) { uint8_t payload[IB_SMP_DATA_SIZE]; ib_smp_t *p_smp; @@ -141,7 +141,7 @@ Exit: * Check that the SM_Key matches. **********************************************************************/ static ib_api_status_t -__osm_sminfo_rcv_check_set_req_legality(IN const ib_smp_t * const p_smp) +sminfo_rcv_check_set_req_legality(IN const ib_smp_t * const p_smp) { ib_sm_info_t *p_smi; @@ -164,8 +164,8 @@ __osm_sminfo_rcv_check_set_req_legality(IN const ib_smp_t * const p_smp) /********************************************************************** **********************************************************************/ static void -__osm_sminfo_rcv_process_set_request(IN osm_sm_t * sm, - IN const osm_madw_t * const p_madw) +sminfo_rcv_process_set_request(IN osm_sm_t * sm, + IN const osm_madw_t * const p_madw) { uint8_t payload[IB_SMP_DATA_SIZE]; ib_smp_t *p_smp; @@ -213,7 +213,7 @@ __osm_sminfo_rcv_process_set_request(IN osm_sm_t * sm, } /* Check the legality of the packet */ - status = __osm_sminfo_rcv_check_set_req_legality(p_smp); + status = sminfo_rcv_check_set_req_legality(p_smp); if (status != IB_SUCCESS) { OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 2F04: " "Check legality failed. AttributeModifier:0x%X RemoteState:%s\n", @@ -306,9 +306,9 @@ Exit: /********************************************************************** **********************************************************************/ static osm_signal_t -__osm_sminfo_rcv_process_get_sm(IN osm_sm_t * sm, - IN const osm_remote_sm_t * const p_sm, - boolean_t light_sweep) +sminfo_rcv_process_get_sm(IN osm_sm_t * sm, + IN const osm_remote_sm_t * const p_sm, + boolean_t light_sweep) { const ib_sm_info_t *p_smi; @@ -339,8 +339,7 @@ __osm_sminfo_rcv_process_get_sm(IN osm_sm_t * sm, break; case IB_SMINFO_STATE_DISCOVERING: case IB_SMINFO_STATE_STANDBY: - if (__osm_sminfo_rcv_remote_sm_is_higher(sm, p_smi) - == TRUE) { + if (sminfo_rcv_remote_sm_is_higher(sm, p_smi) == TRUE) { /* the remote is a higher sm - need to stop sweeping */ sm->master_sm_found = 1; /* save on the sm the guid of the higher SM we found - */ @@ -366,8 +365,7 @@ __osm_sminfo_rcv_process_get_sm(IN osm_sm_t * sm, /* Signal that to the SM state mgr */ osm_sm_state_mgr_signal_master_is_alive(sm); - if (__osm_sminfo_rcv_remote_sm_is_higher(sm, p_smi) == - FALSE) + if (sminfo_rcv_remote_sm_is_higher(sm, p_smi) == FALSE) osm_send_trap144(sm, TRAP_144_MASK_SM_PRIORITY_CHANGE); break; @@ -379,8 +377,8 @@ __osm_sminfo_rcv_process_get_sm(IN osm_sm_t * sm, If we started polling it when it was master, and it moved to standby - then it might be with a lower priority than us - and then we don't want to continue polling it. */ - if (__osm_sminfo_rcv_remote_sm_is_higher - (sm, p_smi) == TRUE) + if (sminfo_rcv_remote_sm_is_higher(sm, p_smi) + == TRUE) osm_sm_state_mgr_signal_master_is_alive(sm); } break; @@ -406,7 +404,7 @@ __osm_sminfo_rcv_process_get_sm(IN osm_sm_t * sm, break; case IB_SMINFO_STATE_STANDBY: if (light_sweep && - __osm_sminfo_rcv_remote_sm_is_higher(sm, p_smi)) + sminfo_rcv_remote_sm_is_higher(sm, p_smi)) sm->p_subn->force_heavy_sweep = TRUE; break; default: @@ -426,8 +424,8 @@ __osm_sminfo_rcv_process_get_sm(IN osm_sm_t * sm, /********************************************************************** **********************************************************************/ static void -__osm_sminfo_rcv_process_get_response(IN osm_sm_t * sm, - IN const osm_madw_t * const p_madw) +sminfo_rcv_process_get_response(IN osm_sm_t * sm, + IN const osm_madw_t * const p_madw) { const ib_smp_t *p_smp; const ib_sm_info_t *p_smi; @@ -508,8 +506,8 @@ __osm_sminfo_rcv_process_get_response(IN osm_sm_t * sm, /* We already know this SM. Update the SMInfo attribute. */ p_sm->smi = *p_smi; - __osm_sminfo_rcv_process_get_sm(sm, p_sm, - osm_madw_get_smi_context_ptr(p_madw)->light_sweep); + sminfo_rcv_process_get_sm(sm, p_sm, + osm_madw_get_smi_context_ptr(p_madw)->light_sweep); _unlock_and_exit: CL_PLOCK_RELEASE(sm->p_lock); @@ -521,8 +519,8 @@ Exit: /********************************************************************** **********************************************************************/ static void -__osm_sminfo_rcv_process_set_response(IN osm_sm_t * sm, - IN const osm_madw_t * const p_madw) +sminfo_rcv_process_set_response(IN osm_sm_t * sm, + IN const osm_madw_t * const p_madw) { const ib_smp_t *p_smp; const ib_sm_info_t *p_smi; @@ -593,16 +591,16 @@ void osm_sminfo_rcv_process(IN void *context, IN void *data) if (p_smi_context->set_method == FALSE) /* this is a response to a Get method */ - __osm_sminfo_rcv_process_get_response(sm, p_madw); + sminfo_rcv_process_get_response(sm, p_madw); else /* this is a response to a Set method */ - __osm_sminfo_rcv_process_set_response(sm, p_madw); + sminfo_rcv_process_set_response(sm, p_madw); } else if (p_smp->method == IB_MAD_METHOD_GET) /* This is a SubnGet request */ - __osm_sminfo_rcv_process_get_request(sm, p_madw); + sminfo_rcv_process_get_request(sm, p_madw); else /* This should be a SubnSet request */ - __osm_sminfo_rcv_process_set_request(sm, p_madw); + sminfo_rcv_process_set_request(sm, p_madw); Exit: OSM_LOG_EXIT(sm->p_log); diff --git a/opensm/opensm/osm_state_mgr.c b/opensm/opensm/osm_state_mgr.c index fe516a6..21ec1f2 100644 --- a/opensm/opensm/osm_state_mgr.c +++ b/opensm/opensm/osm_state_mgr.c @@ -72,7 +72,7 @@ extern osm_signal_t osm_link_mgr_process(IN osm_sm_t * sm, IN uint8_t state); /********************************************************************** **********************************************************************/ -static void __osm_state_mgr_up_msg(IN const osm_sm_t * sm) +static void state_mgr_up_msg(IN const osm_sm_t * sm) { /* * This message should be written only once - when the @@ -89,8 +89,8 @@ static void __osm_state_mgr_up_msg(IN const osm_sm_t * sm) /********************************************************************** **********************************************************************/ -static void __osm_state_mgr_reset_node_count(IN cl_map_item_t * - const p_map_item, IN void *context) +static void state_mgr_reset_node_count(IN cl_map_item_t * const p_map_item, + IN void *context) { osm_node_t *p_node = (osm_node_t *) p_map_item; @@ -99,8 +99,8 @@ static void __osm_state_mgr_reset_node_count(IN cl_map_item_t * /********************************************************************** **********************************************************************/ -static void __osm_state_mgr_reset_port_count(IN cl_map_item_t * - const p_map_item, IN void *context) +static void state_mgr_reset_port_count(IN cl_map_item_t * const p_map_item, + IN void *context) { osm_port_t *p_port = (osm_port_t *) p_map_item; @@ -110,8 +110,8 @@ static void __osm_state_mgr_reset_port_count(IN cl_map_item_t * /********************************************************************** **********************************************************************/ static void -__osm_state_mgr_reset_switch_count(IN cl_map_item_t * const p_map_item, - IN void *context) +state_mgr_reset_switch_count(IN cl_map_item_t * const p_map_item, + IN void *context) { osm_switch_t *p_sw = (osm_switch_t *) p_map_item; @@ -120,8 +120,8 @@ __osm_state_mgr_reset_switch_count(IN cl_map_item_t * const p_map_item, /********************************************************************** **********************************************************************/ -static void __osm_state_mgr_get_sw_info(IN cl_map_item_t * const p_object, - IN void *context) +static void state_mgr_get_sw_info(IN cl_map_item_t * const p_object, + IN void *context) { osm_node_t *p_node; osm_dr_path_t *p_dr_path; @@ -155,8 +155,8 @@ static void __osm_state_mgr_get_sw_info(IN cl_map_item_t * const p_object, Initiate a remote port info request for the given physical port **********************************************************************/ static void -__osm_state_mgr_get_remote_port_info(IN osm_sm_t * sm, - IN osm_physp_t * const p_physp) +state_mgr_get_remote_port_info(IN osm_sm_t * sm, + IN osm_physp_t * const p_physp) { osm_dr_path_t *p_dr_path; osm_dr_path_t rem_node_dr_path; @@ -196,7 +196,7 @@ __osm_state_mgr_get_remote_port_info(IN osm_sm_t * sm, Initiates a thorough sweep of the subnet. Used when there is suspicion that something on the subnet has changed. **********************************************************************/ -static ib_api_status_t __osm_state_mgr_sweep_hop_0(IN osm_sm_t * sm) +static ib_api_status_t state_mgr_sweep_hop_0(IN osm_sm_t * sm) { ib_api_status_t status; osm_dr_path_t dr_path; @@ -221,13 +221,13 @@ static ib_api_status_t __osm_state_mgr_sweep_hop_0(IN osm_sm_t * sm) CL_PLOCK_ACQUIRE(sm->p_lock); cl_qmap_apply_func(&sm->p_subn->node_guid_tbl, - __osm_state_mgr_reset_node_count, sm); + state_mgr_reset_node_count, sm); cl_qmap_apply_func(&sm->p_subn->port_guid_tbl, - __osm_state_mgr_reset_port_count, sm); + state_mgr_reset_port_count, sm); cl_qmap_apply_func(&sm->p_subn->sw_guid_tbl, - __osm_state_mgr_reset_switch_count, sm); + state_mgr_reset_switch_count, sm); /* Set the in_sweep_hop_0 flag in subn to be TRUE. * This will indicate the sweeping not to continue beyond the @@ -259,7 +259,7 @@ static ib_api_status_t __osm_state_mgr_sweep_hop_0(IN osm_sm_t * sm) /********************************************************************** Clear out all existing port lid assignments **********************************************************************/ -static ib_api_status_t __osm_state_mgr_clean_known_lids(IN osm_sm_t * sm) +static ib_api_status_t state_mgr_clean_known_lids(IN osm_sm_t * sm) { ib_api_status_t status = IB_SUCCESS; cl_ptr_vector_t *p_vec = &(sm->p_subn->port_lid_tbl); @@ -283,7 +283,7 @@ static ib_api_status_t __osm_state_mgr_clean_known_lids(IN osm_sm_t * sm) Notifies the transport layer that the local LID has changed, which give it a chance to update address vectors, etc.. **********************************************************************/ -static ib_api_status_t __osm_state_mgr_notify_lid_change(IN osm_sm_t * sm) +static ib_api_status_t state_mgr_notify_lid_change(IN osm_sm_t * sm) { ib_api_status_t status; osm_bind_handle_t h_bind; @@ -319,7 +319,7 @@ Exit: Returns true if the SM port is down. The SM's port object must exist in the port_guid table. **********************************************************************/ -static boolean_t __osm_state_mgr_is_sm_port_down(IN osm_sm_t * sm) +static boolean_t state_mgr_is_sm_port_down(IN osm_sm_t * sm) { ib_net64_t port_guid; osm_port_t *p_port; @@ -370,7 +370,7 @@ Exit: This sets off a "chain reaction" that causes discovery of the subnet. Used when there is suspicion that something on the subnet has changed. **********************************************************************/ -static ib_api_status_t __osm_state_mgr_sweep_hop_1(IN osm_sm_t * sm) +static ib_api_status_t state_mgr_sweep_hop_1(IN osm_sm_t * sm) { ib_api_status_t status = IB_SUCCESS; osm_bind_handle_t h_bind; @@ -516,8 +516,8 @@ static void query_sm_info(cl_map_item_t *item, void *cxt) During a light sweep, check each node to see if the node description is valid and if not issue a ND query. **********************************************************************/ -static void __osm_state_mgr_get_node_desc(IN cl_map_item_t * const p_object, - IN void *context) +static void state_mgr_get_node_desc(IN cl_map_item_t * const p_object, + IN void *context) { osm_madw_context_t mad_context; osm_node_t *const p_node = (osm_node_t *) p_object; @@ -569,7 +569,7 @@ exit: Initiates a lightweight sweep of the subnet. Used during normal sweeps after the subnet is up. **********************************************************************/ -static ib_api_status_t __osm_state_mgr_light_sweep_start(IN osm_sm_t * sm) +static ib_api_status_t state_mgr_light_sweep_start(IN osm_sm_t * sm) { ib_api_status_t status = IB_SUCCESS; osm_bind_handle_t h_bind; @@ -596,11 +596,11 @@ static ib_api_status_t __osm_state_mgr_light_sweep_start(IN osm_sm_t * sm) OSM_LOG_MSG_BOX(sm->p_log, OSM_LOG_VERBOSE, "INITIATING LIGHT SWEEP"); CL_PLOCK_ACQUIRE(sm->p_lock); - cl_qmap_apply_func(p_sw_tbl, __osm_state_mgr_get_sw_info, sm); + cl_qmap_apply_func(p_sw_tbl, state_mgr_get_sw_info, sm); CL_PLOCK_RELEASE(sm->p_lock); CL_PLOCK_ACQUIRE(sm->p_lock); - cl_qmap_apply_func(&sm->p_subn->node_guid_tbl, __osm_state_mgr_get_node_desc, sm); + cl_qmap_apply_func(&sm->p_subn->node_guid_tbl, state_mgr_get_node_desc, sm); CL_PLOCK_RELEASE(sm->p_lock); /* now scan the list of physical ports that were not down but have no remote port */ @@ -628,8 +628,7 @@ static ib_api_status_t __osm_state_mgr_light_sweep_start(IN osm_sm_t * sm) osm_physp_get_dr_path_ptr (p_physp), OSM_LOG_ERROR); - __osm_state_mgr_get_remote_port_info(sm, - p_physp); + state_mgr_get_remote_port_info(sm, p_physp); } } } @@ -649,7 +648,7 @@ _exit: * If there is a remote master SM - return a pointer to it, * else - return NULL. **********************************************************************/ -static osm_remote_sm_t *__osm_state_mgr_exists_other_master_sm(IN osm_sm_t * sm) +static osm_remote_sm_t *state_mgr_exists_other_master_sm(IN osm_sm_t * sm) { cl_qmap_t *p_sm_tbl; osm_remote_sm_t *p_sm; @@ -686,7 +685,7 @@ Exit: * Compare this SM to the local SM. If the local SM is higher - * return NULL, if the remote SM is higher - return a pointer to it. **********************************************************************/ -static osm_remote_sm_t *__osm_state_mgr_get_highest_sm(IN osm_sm_t * sm) +static osm_remote_sm_t *state_mgr_get_highest_sm(IN osm_sm_t * sm) { cl_qmap_t *p_sm_tbl; osm_remote_sm_t *p_sm = NULL; @@ -741,8 +740,8 @@ static osm_remote_sm_t *__osm_state_mgr_get_highest_sm(IN osm_sm_t * sm) * remote_sm indicated. **********************************************************************/ static void -__osm_state_mgr_send_handover(IN osm_sm_t * const sm, - IN osm_remote_sm_t * const p_sm) +state_mgr_send_handover(IN osm_sm_t * const sm, + IN osm_remote_sm_t * const p_sm) { uint8_t payload[IB_SMP_DATA_SIZE]; ib_sm_info_t *p_smi = (ib_sm_info_t *) payload; @@ -811,7 +810,7 @@ Exit: /********************************************************************** * Send Trap 64 on all new ports. **********************************************************************/ -static void __osm_state_mgr_report_new_ports(IN osm_sm_t * sm) +static void state_mgr_report_new_ports(IN osm_sm_t * sm) { ib_gid_t port_gid; ib_mad_notice_attr_t notice; @@ -889,7 +888,7 @@ static void __osm_state_mgr_report_new_ports(IN osm_sm_t * sm) * initialization), but here we'll clean the database from incorrect * information. **********************************************************************/ -static void __osm_state_mgr_check_tbl_consistency(IN osm_sm_t * sm) +static void state_mgr_check_tbl_consistency(IN osm_sm_t * sm) { cl_qmap_t *p_port_guid_tbl; osm_port_t *p_port; @@ -1060,7 +1059,7 @@ static void do_sweep(osm_sm_t * sm) * to do that we want all the ports to be considered * foreign */ - __osm_state_mgr_clean_known_lids(sm); + state_mgr_clean_known_lids(sm); sm->master_sm_found = 0; @@ -1081,7 +1080,7 @@ static void do_sweep(osm_sm_t * sm) && sm->p_subn->force_heavy_sweep == FALSE && sm->p_subn->force_reroute == FALSE && sm->p_subn->subnet_initialization_error == FALSE - && (__osm_state_mgr_light_sweep_start(sm) == IB_SUCCESS)) { + && (state_mgr_light_sweep_start(sm) == IB_SUCCESS)) { if (wait_for_pending_transactions(&sm->p_subn->p_osm->stats)) return; if (!sm->p_subn->force_heavy_sweep) { @@ -1147,12 +1146,12 @@ _repeat_discovery: if (sm->p_subn->sm_state != IB_SMINFO_STATE_MASTER) sm->p_subn->need_update = 1; - status = __osm_state_mgr_sweep_hop_0(sm); + status = state_mgr_sweep_hop_0(sm); if (status != IB_SUCCESS || wait_for_pending_transactions(&sm->p_subn->p_osm->stats)) return; - if (__osm_state_mgr_is_sm_port_down(sm) == TRUE) { + if (state_mgr_is_sm_port_down(sm) == TRUE) { osm_log(sm->p_log, OSM_LOG_SYS, "SM port is down\n"); OSM_LOG_MSG_BOX(sm->p_log, OSM_LOG_VERBOSE, "SM PORT DOWN"); @@ -1164,7 +1163,7 @@ _repeat_discovery: return; } - status = __osm_state_mgr_sweep_hop_1(sm); + status = state_mgr_sweep_hop_1(sm); if (status != IB_SUCCESS || wait_for_pending_transactions(&sm->p_subn->p_osm->stats)) return; @@ -1194,22 +1193,21 @@ _repeat_discovery: * see if it is higher than our local sm. */ if (sm->p_subn->sm_state == IB_SMINFO_STATE_MASTER) { - p_remote_sm = __osm_state_mgr_get_highest_sm(sm); + p_remote_sm = state_mgr_get_highest_sm(sm); if (p_remote_sm != NULL) { /* report new ports (trap 64) before leaving MASTER */ - __osm_state_mgr_report_new_ports(sm); + state_mgr_report_new_ports(sm); /* need to handover the mastership * to the remote sm, and move to standby */ - __osm_state_mgr_send_handover(sm, p_remote_sm); + state_mgr_send_handover(sm, p_remote_sm); osm_sm_state_mgr_process(sm, OSM_SM_SIGNAL_HANDOVER_SENT); return; } else { /* We are the highest sm - check to see if there is * a remote SM that is in master state. */ - p_remote_sm = - __osm_state_mgr_exists_other_master_sm(sm); + p_remote_sm = state_mgr_exists_other_master_sm(sm); if (p_remote_sm != NULL) { /* There is a remote SM that is master. * need to wait for that SM to relinquish control @@ -1255,7 +1253,7 @@ _repeat_discovery: OSM_LOG_MSG_BOX(sm->p_log, OSM_LOG_VERBOSE, "SM LID ASSIGNMENT COMPLETE - STARTING SUBNET LID CONFIG"); - __osm_state_mgr_notify_lid_change(sm); + state_mgr_notify_lid_change(sm); osm_lid_mgr_process_subnet(&sm->lid_mgr); if (wait_for_pending_transactions(&sm->p_subn->p_osm->stats)) @@ -1265,7 +1263,7 @@ _repeat_discovery: * the port_lid_tbl under the subnet. There might be * errors in it if PortInfo Set requests didn't reach * their destination. */ - __osm_state_mgr_check_tbl_consistency(sm); + state_mgr_check_tbl_consistency(sm); OSM_LOG_MSG_BOX(sm->p_log, OSM_LOG_VERBOSE, "LID ASSIGNMENT COMPLETE - STARTING SWITCH TABLE CONFIG"); @@ -1333,7 +1331,7 @@ _repeat_discovery: /* * Send trap 64 on newly discovered endports */ - __osm_state_mgr_report_new_ports(sm); + state_mgr_report_new_ports(sm); /* in any case we zero this flag */ sm->p_subn->coming_out_of_standby = FALSE; @@ -1347,7 +1345,7 @@ _repeat_discovery: } else { sm->p_subn->need_update = 0; osm_dump_all(sm->p_subn->p_osm); - __osm_state_mgr_up_msg(sm); + state_mgr_up_msg(sm); sm->p_subn->first_time_master_sweep = FALSE; if (osm_log_is_active(sm->p_log, OSM_LOG_VERBOSE)) diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c index fe0a446..20be5d3 100644 --- a/opensm/opensm/osm_ucast_mgr.c +++ b/opensm/opensm/osm_ucast_mgr.c @@ -107,8 +107,7 @@ osm_ucast_mgr_init(IN osm_ucast_mgr_t * const p_mgr, IN osm_sm_t * sm) Add each switch's own and neighbor LIDs to its LID matrix **********************************************************************/ static void -__osm_ucast_mgr_process_hop_0_1(IN cl_map_item_t * const p_map_item, - IN void *context) +ucast_mgr_process_hop_0_1(IN cl_map_item_t * const p_map_item, IN void *context) { osm_switch_t *const p_sw = (osm_switch_t *) p_map_item; osm_node_t *p_remote_node; @@ -135,11 +134,11 @@ __osm_ucast_mgr_process_hop_0_1(IN cl_map_item_t * const p_map_item, /********************************************************************** **********************************************************************/ static void -__osm_ucast_mgr_process_neighbor(IN osm_ucast_mgr_t * const p_mgr, - IN osm_switch_t * const p_this_sw, - IN osm_switch_t * const p_remote_sw, - IN const uint8_t port_num, - IN const uint8_t remote_port_num) +ucast_mgr_process_neighbor(IN osm_ucast_mgr_t * const p_mgr, + IN osm_switch_t * const p_this_sw, + IN osm_switch_t * const p_remote_sw, + IN const uint8_t port_num, + IN const uint8_t remote_port_num) { osm_switch_t *p_sw; cl_map_item_t *item; @@ -204,10 +203,10 @@ find_and_add_remote_sys(osm_switch_t *sw, uint8_t port, } static void -__osm_ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, - IN osm_switch_t * const p_sw, - IN osm_port_t * const p_port, - IN unsigned lid_offset) +ucast_mgr_process_port(IN osm_ucast_mgr_t * const p_mgr, + IN osm_switch_t * const p_sw, + IN osm_port_t * const p_port, + IN unsigned lid_offset) { uint16_t min_lid_ho; uint16_t max_lid_ho; @@ -472,8 +471,8 @@ static void free_ports_priv(osm_ucast_mgr_t *mgr) } static void -__osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, - IN void *context) +ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, + IN void *context) { osm_ucast_mgr_t *p_mgr = context; osm_switch_t *const p_sw = (osm_switch_t *) p_map_item; @@ -504,7 +503,7 @@ __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, for (item = cl_qlist_head(list); item != cl_qlist_end(list); item = cl_qlist_next(item)) { osm_port_t *port = cl_item_obj(item, port, list_item); - __osm_ucast_mgr_process_port(p_mgr, p_sw, port, i); + ucast_mgr_process_port(p_mgr, p_sw, port, i); } } @@ -519,8 +518,8 @@ __osm_ucast_mgr_process_tbl(IN cl_map_item_t * const p_map_item, /********************************************************************** **********************************************************************/ static void -__osm_ucast_mgr_process_neighbors(IN cl_map_item_t * const p_map_item, - IN void *context) +ucast_mgr_process_neighbors(IN cl_map_item_t * const p_map_item, + IN void *context) { osm_switch_t *const p_sw = (osm_switch_t *) p_map_item; osm_ucast_mgr_t *const p_mgr = (osm_ucast_mgr_t *) context; @@ -560,10 +559,10 @@ __osm_ucast_mgr_process_neighbors(IN cl_map_item_t * const p_map_item, if (!p_physp || !osm_link_is_healthy(p_physp)) continue; - __osm_ucast_mgr_process_neighbor(p_mgr, p_sw, - p_remote_node->sw, - (uint8_t) port_num, - remote_port_num); + ucast_mgr_process_neighbor(p_mgr, p_sw, + p_remote_node->sw, + (uint8_t) port_num, + remote_port_num); } } @@ -659,8 +658,7 @@ int osm_ucast_mgr_build_lid_matrices(IN osm_ucast_mgr_t * const p_mgr) Set the switch matrices for each switch's own port 0 LID(s) then set the lid matrices for the each switch's leaf nodes. */ - cl_qmap_apply_func(p_sw_guid_tbl, - __osm_ucast_mgr_process_hop_0_1, p_mgr); + cl_qmap_apply_func(p_sw_guid_tbl, ucast_mgr_process_hop_0_1, p_mgr); /* Get the switch matrices for each switch's neighbors. @@ -701,8 +699,7 @@ int osm_ucast_mgr_build_lid_matrices(IN osm_ucast_mgr_t * const p_mgr) i++) { p_mgr->some_hop_count_set = FALSE; cl_qmap_apply_func(p_sw_guid_tbl, - __osm_ucast_mgr_process_neighbors, - p_mgr); + ucast_mgr_process_neighbors, p_mgr); } OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, "Min-hop propagated in %d steps\n", i); @@ -905,8 +902,8 @@ static int ucast_mgr_build_lfts(osm_ucast_mgr_t *p_mgr) cl_qmap_apply_func(&p_mgr->p_subn->port_guid_tbl, add_port_to_order_list, p_mgr); - cl_qmap_apply_func(&p_mgr->p_subn->sw_guid_tbl, - __osm_ucast_mgr_process_tbl, p_mgr); + cl_qmap_apply_func(&p_mgr->p_subn->sw_guid_tbl, ucast_mgr_process_tbl, + p_mgr); cl_qlist_remove_all(&p_mgr->port_order_list); diff --git a/opensm/opensm/osm_vl15intf.c b/opensm/opensm/osm_vl15intf.c index 0703a4f..48a06af 100644 --- a/opensm/opensm/osm_vl15intf.c +++ b/opensm/opensm/osm_vl15intf.c @@ -109,7 +109,7 @@ static void vl15_send_mad(osm_vl15_t * p_vl, osm_madw_t * p_madw) cl_atomic_dec(&p_vl->p_stats->qp0_unicasts_sent); } -static void __osm_vl15_poller(IN void *p_ptr) +static void vl15_poller(IN void *p_ptr) { ib_api_status_t status; osm_madw_t *p_madw; @@ -272,7 +272,7 @@ osm_vl15_init(IN osm_vl15_t * const p_vl, Initialize the thread after all other dependent objects have been initialized. */ - status = cl_thread_init(&p_vl->poller, __osm_vl15_poller, p_vl, + status = cl_thread_init(&p_vl->poller, vl15_poller, p_vl, "opensm poller"); if (status != IB_SUCCESS) goto Exit; From keshetti.mahesh at gmail.com Mon Mar 23 04:39:22 2009 From: keshetti.mahesh at gmail.com (Keshetti Mahesh) Date: Mon, 23 Mar 2009 17:09:22 +0530 Subject: [ofa-general] ***SPAM*** Re: IPoIB problem In-Reply-To: <829ded920903200215vfe2618fwe584703d7b62c3d3@mail.gmail.com> References: <829ded920903200215vfe2618fwe584703d7b62c3d3@mail.gmail.com> Message-ID: <829ded920903230439y41c06b2q374802ec1a00d103@mail.gmail.com> Sorry to bother you all. It was actually a problem related with 'iptables'. And it is solved now. Thanks, Mahesh On Fri, Mar 20, 2009 at 2:45 PM, Keshetti Mahesh wrote: > Hello list, > > I have isolated an enclosure consisting of 4 nodes each with > a Mellanox HCA from an IB cluster containing multiple such > enclosures. And connected nodes of the isolated enclosure > with another IB switch forming a different subnet. This enclosure > uses a same "ethernet" network which is used for the main cluster. > > Everything (IB related) works fine in the isolated cluster if I don't > use IPoIB. When I load IPoIB and run any application using > TCP/IP (e.g. ping) on IPoIB interface whole system hangs for 10-15 > minutes and strangely all IB related applications (e.g OpenSM) gets killed. > ping to the local IPoIB interface works. But ping to any remote IPoIB > interface results into above situation. > > Can anyone please tell me is there anything obviously wrong with the above > described configuration? Any help on resolving this issue is greatly > appreciated. > > -Mahesh > From jon at opengridcomputing.com Mon Mar 23 07:46:59 2009 From: jon at opengridcomputing.com (Jon Mason) Date: Mon, 23 Mar 2009 09:46:59 -0500 Subject: [ofa-general] Re: ofed-1.4.1-rc2 NFS-RDMA server crash, In-Reply-To: <49C42424.6050103@mellanox.com> References: <49C42424.6050103@mellanox.com> Message-ID: <20090323144657.GA9840@opengridcomputing.com> On Fri, Mar 20, 2009 at 04:17:56PM -0700, Vu Pham wrote: > Hi Jon, > > I ran connectathon test -N100 and get this crash on the server. Both > server/client are RHEL 5.2 x64 with connectX HCAs > > Should I open a bug# on bugzilla? If you hit a bug, you should open one in bugzilla so it can be tracked. Do you see the same behavior on mainline or is this isolated to the RHEL5.2 backport? Thanks, Jon > > thanks, > -vu > Mar 20 10:05:34 vlab-007 kernel: EXT3-fs: mounted filesystem with ordered data m > ode. > general protection fault: 0000 [1] svcrdma: error fast registering xdr for xprt > ffff81022e4f0c00SMP > last sysfs file: /devices/pci0000:00/0000:00:00.0/local_cpus > CPU 4 > Modules linked in: svcrdma(U) nfsd(U) lockd(U) nfs_acl(U) auth_rpcgss(U) exportf > s(U) autofs4 hidp rfcomm l2cap bluetooth sunrpc(U) rdma_ucm(U) rdma_cm(U) iw_cm( > U) ib_addr(U) ib_ipoib(U) ipoib_helper(U) ib_cm(U) ib_sa(U) ipv6 xfrm_nalgo cryp > to_api ib_uverbs(U) ib_umad(U) mlx4_ib(U) dm_mirror dm_multipath dm_mod raid0 vi > deo sbs backlight i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc > lp parport i2c_i801 mlx4_core(U) e1000e serio_raw pcspkr ib_mthca(U) shpchp i2c_ > core ib_mad(U) ib_core(U) sg ata_piix libata mptsas mptscsih mptbase scsi_transp > ort_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd > Pid: 0, comm: swapper Tainted: G 2.6.18-92.el5 #1 > RIP: 0010:[] [] mark_clean+0x50/0x77 > RSP: 0018:ffff81022fc6fe18 EFLAGS: 00010202 > RAX: 5b98396687b9ba94 RBX: ffff81022349d0c0 RCX: 0000000000000080 > RDX: 0000140d41402000 RSI: 0140d41402000000 RDI: 0140551402001000 > RBP: 0140d41402000000 R08: 0140551402001000 R09: 5b98396687b9be94 > R10: ffff81022fd68038 R11: ffffffff800928d3 R12: ffff81022e4f0c00 > R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff803c82e0 > FS: 0000000000000000(0000) GS:ffff81022fc20d40(0000) knlGS:0000000000000000 > CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > CR2: 00002b5f92237000 CR3: 0000000215470000 CR4: 00000000000006e0 > Process swapper (pid: 0, threadinfo ffff81022fc66000, task ffff81022fc21080) > Stack: ffffffff885a45da ffff81022e4f0e60 ffff81022e4f0c00 ffff8102150d6140 > ffff8102152ca1c0 ffff81022c7e6600 ffffffff885a4a8f ffff8102150d6140 > ffffffff00000004 0000000000000032 ffff81022a338200 ffff81022a338af0 > Call Trace: > [] :svcrdma:svc_rdma_put_frmr+0xbc/0x117 > [] :svcrdma:sq_cq_reap+0x11a/0x1a8 > [] _spin_lock_bh+0x9/0x14 > [] :svcrdma:dto_tasklet_func+0x13a/0x17a > [] :mlx4_core:mlx4_eq_int+0x27e/0x28f > [] tasklet_action+0x62/0xac > [] __do_softirq+0x5e/0xd6 > [] end_msi_irq_w_maskbit+0xf/0x1c > [] call_softirq+0x1c/0x28 > [] do_softirq+0x2c/0x85 > [] do_IRQ+0xec/0xf5 > [] mwait_idle+0x0/0x4a > [] ret_from_intr+0x0/0xa > [] mwait_idle+0x36/0x4a > [] cpu_idle+0x95/0xb8 > [] start_secondary+0x45a/0x469 > > > Code: 49 8b 01 48 6b d2 38 48 83 e0 fc 48 01 d0 f0 0f ba 28 09 49 > RIP [] mark_clean+0x50/0x77 > RSP From nicolas.morey-chaisemartin at ext.bull.net Mon Mar 23 08:22:15 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Mon, 23 Mar 2009 16:22:15 +0100 Subject: [ofa-general] [PATCH] Enhance min hops counters in Ftree Message-ID: <49C7A927.1040304@ext.bull.net> This patch enhances the use of the min hop table done in the Fat-Tree algorithm. Before this patch, the algorithm was using the osm_sw hops table to store the minhop values toward any lid (Switch or not). As this table is allocated as we need it, it required a lot of malloc calls and quite some time to set the hops values on remote ports. This patch corrects this behaviour: -The osm_sw hops table is only used for switch lid -ftree_sw_t struct now has its own hop table (only 1 dimensionnal as we don't need to know which port is used) to store its minhop value Signed-off-by: Nicolas Morey-Chaisemartin --- Here are some performances tests I have realized with and without this patch: For ~3500 nodes: Without = 0.981s / 169.5MB used memory With = 0.549s / 123MB used memory For over 5000 nodes: Without = 2.25s / 308MB used memory With = 1.29s / 186MB used memory Computation time is taken in do_ouring function (thus only taking computation not LFT setting time or anything else) opensm/opensm/osm_ucast_ftree.c | 68 +++++++++++++++++++++++++++++++-------- 1 files changed, 54 insertions(+), 14 deletions(-) diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c index 58d1c14..a5da571 100644 --- a/opensm/opensm/osm_ucast_ftree.c +++ b/opensm/opensm/osm_ucast_ftree.c @@ -172,6 +172,7 @@ typedef struct ftree_sw_t_ { uint8_t up_port_groups_num; boolean_t is_leaf; unsigned down_port_groups_idx; + uint8_t *hops; } ftree_sw_t; /*************************************************** @@ -553,6 +554,9 @@ static ftree_sw_t *sw_create(IN ftree_fabric_t * p_ftree, /* initialize lft buffer */ memset(p_osm_sw->new_lft, OSM_NO_PATH, IB_LID_UCAST_END_HO + 1); + p_sw->hops = + malloc(cl_ntoh16(p_osm_sw->max_lid_ho) * sizeof(*(p_sw->hops))); + memset(p_sw->hops, OSM_NO_PATH, cl_ntoh16(p_osm_sw->max_lid_ho)); return p_sw; } /* sw_create() */ @@ -565,6 +569,7 @@ static void sw_destroy(IN ftree_fabric_t * p_ftree, IN ftree_sw_t * p_sw) if (!p_sw) return; + free(p_sw->hops); for (i = 0; i < p_sw->down_port_groups_num; i++) port_group_destroy(p_sw->down_port_groups[i]); @@ -692,32 +697,56 @@ static void sw_add_port(IN ftree_sw_t * p_sw, IN uint8_t port_num, /***************************************************/ static inline cl_status_t sw_set_hops(IN ftree_sw_t * p_sw, IN uint16_t lid_ho, - IN uint8_t port_num, IN uint8_t hops) + IN uint8_t port_num, IN uint8_t hops, + IN boolean_t is_target_sw) { /* set local min hop table(LID) */ - return osm_switch_set_hops(p_sw->p_osm_sw, lid_ho, port_num, hops); + p_sw->hops[lid_ho] = hops; + if (is_target_sw) + return osm_switch_set_hops(p_sw->p_osm_sw, lid_ho, port_num, + hops); + return 0; } /***************************************************/ static int set_hops_on_remote_sw(IN ftree_port_group_t * p_group, - IN ib_net16_t target_lid, IN uint8_t hops) + IN ib_net16_t target_lid, IN uint8_t hops, + IN boolean_t is_target_sw) { ftree_port_t *p_port; uint8_t i, ports_num; ftree_sw_t *p_remote_sw = p_group->remote_hca_or_sw.p_sw; - + /* if lid is a switch, we set the min hop table in the osm_switch struct */ CL_ASSERT(p_group->remote_node_type == IB_NODE_TYPE_SWITCH); + p_remote_sw->hops[cl_ntoh16(target_lid)] = hops; + + /* If taget lid is a switch we set the min hop table values + * for each port on the associated osm_sw struct */ + if (!is_target_sw) + return 0; + ports_num = (uint8_t) cl_ptr_vector_get_size(&p_group->ports); for (i = 0; i < ports_num; i++) { cl_ptr_vector_at(&p_group->ports, i, (void *)&p_port); if (sw_set_hops(p_remote_sw, cl_ntoh16(target_lid), - p_port->remote_port_num, hops)) + p_port->remote_port_num, hops, is_target_sw)) return -1; } return 0; } +/***************************************************/ + +static inline uint8_t +sw_get_least_hops(IN ftree_sw_t * p_sw, IN ib_net16_t target_lid) +{ + CL_ASSERT(cl_ntoh16(target_lid) < + cl_ntoh16(p_sw->p_osm_sw->max_lid_ho)); + CL_ASSERT(p_sw->hops != NULL); + return p_sw->hops[cl_ntoh16(target_lid)]; +} + /*************************************************** ** ** ftree_hca_t functions @@ -1883,6 +1912,7 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, IN uint8_t target_rank, IN boolean_t is_real_lid, IN boolean_t is_main_path, + IN boolean_t is_target_a_sw, IN uint8_t highest_rank_in_route, IN uint16_t reverse_hops) { @@ -1945,9 +1975,7 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, set LFT(target_lid) on the remote switch to the remote port */ p_remote_sw = p_group->remote_hca_or_sw.p_sw; - if (osm_switch_get_least_hops(p_remote_sw->p_osm_sw, - cl_ntoh16(target_lid)) != - OSM_NO_PATH) { + if (sw_get_least_hops(p_remote_sw, target_lid) != OSM_NO_PATH) { /* Loop in the fabric - we already routed the remote switch on our way UP, and now we see it again on our way DOWN */ OSM_LOG(&p_ftree->p_osm->log, OSM_LOG_DEBUG, @@ -2013,7 +2041,8 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, highest_rank_in_route) + (p_remote_sw->rank - highest_rank_in_route) + - reverse_hops * 2)); + reverse_hops * 2), + is_target_a_sw); } /* The number of upgoing routes is tracked in the @@ -2032,6 +2061,7 @@ fabric_route_upgoing_by_going_down(IN ftree_fabric_t * p_ftree, target_rank, /* rank of the LID that we're routing to */ is_real_lid, /* whether the target LID is real or dummy */ is_main_path, /* whether this is path to HCA that should by tracked by counters */ + is_target_a_sw, /* Wheter target lid is a switch or not */ highest_rank_in_route, reverse_hops); /* highest visited point in the tree before going down */ } /* done scanning all the down-going port groups */ @@ -2066,6 +2096,7 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, IN uint8_t target_rank, IN boolean_t is_real_lid, IN boolean_t is_main_path, + IN boolean_t is_target_a_sw, IN uint16_t reverse_hop_credit, IN uint16_t reverse_hops) { @@ -2088,6 +2119,7 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, target_rank, /* rank of the LID that we're routing to */ is_real_lid, /* whether this target LID is real or dummy */ is_main_path, /* whether this path to HCA should by tracked by counters */ + is_target_a_sw, /* Wheter target lid is a switch or not */ p_sw->rank, /* the highest visited point in the tree before going down */ reverse_hops); /* Number of reverse_hops done up to this point */ @@ -2117,6 +2149,7 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, target_rank, /* rank of the LID that we're routing to */ is_real_lid, /* whether this target LID is real or dummy */ is_main_path, /* whether this is path to HCA that should by tracked by counters */ + is_target_a_sw, /* Wheter target lid is a switch or not */ reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ reverse_hops + 1); /* Number of reverse_hops done up to this point */ } @@ -2227,7 +2260,7 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, set_hops_on_remote_sw(p_min_group, target_lid, target_rank - p_remote_sw->rank + - 2 * reverse_hops); + 2 * reverse_hops, is_target_a_sw); } /* Recursion step: @@ -2238,6 +2271,7 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, target_rank, /* rank of the LID that we're routing to */ is_real_lid, /* whether this target LID is real or dummy */ is_main_path, /* whether this is path to HCA that should by tracked by counters */ + is_target_a_sw, /* Wheter target lid is a switch or not */ reverse_hop_credit, /* Remaining reverse_hops allowed */ reverse_hops); /* Number of reverse_hops done up to this point */ } @@ -2310,7 +2344,7 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, set_hops_on_remote_sw(p_group, target_lid, target_rank - p_remote_sw->rank + - 2 * reverse_hops); + 2 * reverse_hops, is_target_a_sw); /* Recursion step: Assign downgoing ports by stepping up, starting on REMOTE switch. */ @@ -2320,6 +2354,7 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, target_rank, /* rank of the LID that we're routing to */ TRUE, /* whether the target LID is real or dummy */ FALSE, /* whether this is path to HCA that should by tracked by counters */ + is_target_a_sw, /* Wheter target lid is a switch or not */ reverse_hop_credit, /* Remaining reverse_hops allowed */ reverse_hops); /* Number of reverse_hops done up to this point */ } @@ -2348,6 +2383,7 @@ static void fabric_route_downgoing_by_going_up(IN ftree_fabric_t * p_ftree, target_rank, /* rank of the LID that we're routing to */ TRUE, /* whether the target LID is real or dummy */ TRUE, /* whether this is path to HCA that should by tracked by counters */ + is_target_a_sw, /* Wheter target lid is a switch or not */ reverse_hop_credit - 1, /* Remaining reverse_hops allowed */ reverse_hops + 1); /* Number of reverse_hops done up to this point */ } @@ -2427,7 +2463,7 @@ static void fabric_route_to_cns(IN ftree_fabric_t * p_ftree) /* set local min hop table(LID) to route to the CA */ sw_set_hops(p_sw, cl_ntoh16(hca_lid), - p_port->port_num, 1); + p_port->port_num, 1, FALSE); /* Assign downgoing ports by stepping up. Since we're routing here only CNs, we're routing it as REAL @@ -2438,6 +2474,7 @@ static void fabric_route_to_cns(IN ftree_fabric_t * p_ftree) p_sw->rank + 1, /* rank of the LID that we're routing to */ TRUE, /* whether this HCA LID is real or dummy */ TRUE, /* whether this path to HCA should by tracked by counters */ + FALSE, /* wheter target lid is a switch or not */ 0, /* Number of reverse hops allowed */ 0); /* Number of reverse hops done yet */ @@ -2465,6 +2502,7 @@ static void fabric_route_to_cns(IN ftree_fabric_t * p_ftree) 0, /* rank of the LID that we're routing to - ignored for dummy HCA */ FALSE, /* whether this HCA LID is real or dummy */ TRUE, /* whether this path to HCA should by tracked by counters */ + FALSE, /* Wheter the target LID is a switch or not */ 0, /* Number of reverse hops allowed */ 0); /* Number of reverse hops done yet */ } @@ -2540,7 +2578,7 @@ static void fabric_route_to_non_cns(IN ftree_fabric_t * p_ftree) /* set local min hop table(LID) to route to the CA */ sw_set_hops(p_sw, cl_ntoh16(hca_lid), port_num_on_switch, /* port num */ - 1); /* hops */ + 1, FALSE); /* hops */ /* Assign downgoing ports by stepping up. We're routing REAL targets. They are not CNs and not included @@ -2552,6 +2590,7 @@ static void fabric_route_to_non_cns(IN ftree_fabric_t * p_ftree) p_sw->rank + 1, /* rank of the LID that we're routing to */ TRUE, /* whether this HCA LID is real or dummy */ TRUE, /* whether this path to HCA should by tracked by counters */ + FALSE, /* Wheter the target LID is a switch or not */ p_hca_port_group->is_io ? p_ftree->p_osm->subn.opt.max_reverse_hops : 0, /* Number or reverse hops allowed */ 0); /* Number or reverse hops done yet */ } @@ -2596,7 +2635,7 @@ static void fabric_route_to_switches(IN ftree_fabric_t * p_ftree) /* set min hop table of the switch to itself */ sw_set_hops(p_sw, cl_ntoh16(p_sw->base_lid), 0, /* port_num */ - 0); /* hops */ + 0, TRUE); /* hops */ fabric_route_downgoing_by_going_up(p_ftree, p_sw, /* local switch - used as a route-downgoing alg. start point */ NULL, /* prev. position switch */ @@ -2604,6 +2643,7 @@ static void fabric_route_to_switches(IN ftree_fabric_t * p_ftree) p_sw->rank, /* rank of the LID that we're routing to */ TRUE, /* whether the target LID is a real or dummy */ FALSE, /* whether this path to HCA should by tracked by counters */ + TRUE, /* Wheter the target LID is a switch or not */ 0, /* Number of reverse hops allowed */ 0); /* Number of reverse hops done yet */ } -- 1.6.2-rc2.GIT From vuhuong at mellanox.com Mon Mar 23 09:55:02 2009 From: vuhuong at mellanox.com (Vu Pham) Date: Mon, 23 Mar 2009 09:55:02 -0700 Subject: [ofa-general][PATCH] NFS-RDMA fix for connectX fast registration wr, In-Reply-To: <49c6ca6b.29578c0a.294a.ffffcd1a@mx.google.com> References: <49C422FC.7040303@mellanox.com> <49c6b719.0e538c0a.3e06.ffff8ed8@mx.google.com> <49c6ca6b.29578c0a.294a.ffffcd1a@mx.google.com> Message-ID: <49C7BEE6.7020907@mellanox.com> Tom, > Vu - I noticed this wasn't a usual kernel patch, so I've recoded it > as a patch against Trond's tree, where I have the connectathon > patches queued too. I can send it to him with your signed-off-by, > if that's ok. > > Here is the kernel patch. It's tested with latest connectX FW (2.6.636) > I do want to be sure the patch is tested though. Does it work on > earlier COnnectX firmware that may not support FRMR? I assume > that fw doesn't set the bits the verbs.c file is looking for to detect > FRMR support... correct? > The patch does not work with old FW. The old fw set the bit; however, it has bug on FRWR thanks, -vu > Tom. > > At 06:09 PM 3/22/2009, Tom Talpey wrote: > >> At 02:27 PM 3/22/2009, James Lentini wrote: >> >>> On Fri, 20 Mar 2009, Vu Pham wrote: >>> >>> >>>> Hi Jon, >>>> >>>> Here is the patch for the file >>>> kernel_patches/fixes/nfsrdma_to_2_6_28_18.patch. This patch will >>>> >> fix the fast >> >>>> registration wr for Mellanox ConnectX. >>>> >>>> ConnectX FRMR requires local write enable together with remote rdma write >>>> enable >>>> >>>> thanks, >>>> -vu >>>> >>>> >>> Vu, >>> >>> Are you going to submit this fix upstream? >>> >> Agreed - if setting this bit fixes the FRMR issue, then it should go to >> Trond asap for inclusion in 2.6.30 (hopefully). Can someone verify that >> it works? I don't have access to ConnectX hardware as I am at the >> OFA workshop. >> >> Tom. >> >> >>> -james >>> >>> >>> >>>> --- >>>> >>> ofa_kernel-1.4.1.config/kernel_patches/fixes/nfsrdma_to_2.6.28_13.patc >>> h 2009-03-19 08:42:07.000000000 -0700 >>> >>>> +++ ofa_kernel-1.4.1/kernel_patches/fixes/nfsrdma_to_2.6.28_13.patch >>>> >>> 2009-03-19 08:49:22.000000000 -0700 >>> >>>> @@ -167,7 +167,7 @@ >>>> case RPCRDMA_MTHCAFMR: >>>> case RPCRDMA_MEMWINDOWS_ASYNC: >>>> case RPCRDMA_MEMWINDOWS: >>>> -@@ -1391,6 +1457,96 @@ rpcrdma_unmap_one(struct rpcrdma_ia *ia, >>>> >>> struct rpcrdma_mr_seg *seg) >>> >>>> +@@ -1391,6 +1457,98 @@ rpcrdma_unmap_one(struct rpcrdma_ia *ia, >>>> >>> struct rpcrdma_mr_seg *seg) >>> >>>> } >>>> >>>> static int >>>> @@ -214,8 +214,10 @@ >>>> + frmr_wr.wr.fast_reg.page_list_len = i; >>>> + frmr_wr.wr.fast_reg.page_shift = PAGE_SHIFT; >>>> + frmr_wr.wr.fast_reg.length = i << PAGE_SHIFT; >>>> -+ frmr_wr.wr.fast_reg.access_flags = (writing ? >>>> -+ IB_ACCESS_REMOTE_WRITE : IB_ACCESS_REMOTE_READ); >>>> ++ frmr_wr.wr.fast_reg.access_flags = >>>> ++ (writing ? >>>> ++ (IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE) : >>>> ++ IB_ACCESS_REMOTE_READ); >>>> + frmr_wr.wr.fast_reg.rkey = seg1->mr_chunk.rl_mw->r.frmr.fr_mr->rkey; >>>> + DECR_CQCOUNT(&r_xprt->rx_ep); >>>> + >>>> > > -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: enable_local_write_connectX.patch URL: From john.russo at qlogic.com Mon Mar 23 10:54:04 2009 From: john.russo at qlogic.com (John Russo) Date: Mon, 23 Mar 2009 12:54:04 -0500 Subject: [ofa-general] ***SPAM*** Issues with OFED 1.4 & AMD based HP Proliant Servers Message-ID: We are seeing issues running the IMB-MPI1 test with 24 processes. We see compute nodes hanging during the test. The only systems that exhibit the issue are the AMD Based HP Proliant systems. Before I open a PR, I wanted to see if anyone else has ever tested with these systems and if so, did you have any similar issues? [cid:image001.jpg at 01C9ABBE.D2476500] __________________________ John F. Russo Manager, Engineering QLogic Corporation 780 Fifth Avenue, Suite 140 King of Prussia, PA 19406 Direct: 610-233-4866 Main: 610-233-4800 Fax: 610-233-4777 Cell: 610-246-9903 Email: John.Russo at qlogic.com www.qlogic.com True success is the undeniable truth that we have proved ourselves. -Joe Luppino-Esposito -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 3677 bytes Desc: image001.jpg URL: From vuhuong at mellanox.com Mon Mar 23 11:06:59 2009 From: vuhuong at mellanox.com (Vu Pham) Date: Mon, 23 Mar 2009 11:06:59 -0700 Subject: [ofa-general] Re: ofed-1.4.1-rc2 NFS-RDMA server crash, In-Reply-To: <20090323144657.GA9840@opengridcomputing.com> References: <49C42424.6050103@mellanox.com> <20090323144657.GA9840@opengridcomputing.com> Message-ID: <49C7CFC3.9090106@mellanox.com> Jon Mason wrote: > On Fri, Mar 20, 2009 at 04:17:56PM -0700, Vu Pham wrote: > >> Hi Jon, >> >> I ran connectathon test -N100 and get this crash on the server. Both >> server/client are RHEL 5.2 x64 with connectX HCAs >> >> Should I open a bug# on bugzilla? >> > > If you hit a bug, you should open one in bugzilla so it can be tracked. > OK - I just open bug #1571 > Do you see the same behavior on mainline or is this isolated to the > RHEL5.2 backport? > I run server on mainline kernel 2.6.27. The server fail at same place; however, it does not crash general protection fault: 0000 [1] svcrdma: error fast registering xdr for xprt ffff81022e4f0c00SMP thanks, -vu > Thanks, > Jon > > >> thanks, >> -vu >> > > >> Mar 20 10:05:34 vlab-007 kernel: EXT3-fs: mounted filesystem with ordered data m >> ode. >> general protection fault: 0000 [1] svcrdma: error fast registering xdr for xprt >> ffff81022e4f0c00SMP >> last sysfs file: /devices/pci0000:00/0000:00:00.0/local_cpus >> CPU 4 >> Modules linked in: svcrdma(U) nfsd(U) lockd(U) nfs_acl(U) auth_rpcgss(U) exportf >> s(U) autofs4 hidp rfcomm l2cap bluetooth sunrpc(U) rdma_ucm(U) rdma_cm(U) iw_cm( >> U) ib_addr(U) ib_ipoib(U) ipoib_helper(U) ib_cm(U) ib_sa(U) ipv6 xfrm_nalgo cryp >> to_api ib_uverbs(U) ib_umad(U) mlx4_ib(U) dm_mirror dm_multipath dm_mod raid0 vi >> deo sbs backlight i2c_ec button battery asus_acpi acpi_memhotplug ac parport_pc >> lp parport i2c_i801 mlx4_core(U) e1000e serio_raw pcspkr ib_mthca(U) shpchp i2c_ >> core ib_mad(U) ib_core(U) sg ata_piix libata mptsas mptscsih mptbase scsi_transp >> ort_sas sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd >> Pid: 0, comm: swapper Tainted: G 2.6.18-92.el5 #1 >> RIP: 0010:[] [] mark_clean+0x50/0x77 >> RSP: 0018:ffff81022fc6fe18 EFLAGS: 00010202 >> RAX: 5b98396687b9ba94 RBX: ffff81022349d0c0 RCX: 0000000000000080 >> RDX: 0000140d41402000 RSI: 0140d41402000000 RDI: 0140551402001000 >> RBP: 0140d41402000000 R08: 0140551402001000 R09: 5b98396687b9be94 >> R10: ffff81022fd68038 R11: ffffffff800928d3 R12: ffff81022e4f0c00 >> R13: 0000000000000000 R14: 0000000000000000 R15: ffffffff803c82e0 >> FS: 0000000000000000(0000) GS:ffff81022fc20d40(0000) knlGS:0000000000000000 >> CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b >> CR2: 00002b5f92237000 CR3: 0000000215470000 CR4: 00000000000006e0 >> Process swapper (pid: 0, threadinfo ffff81022fc66000, task ffff81022fc21080) >> Stack: ffffffff885a45da ffff81022e4f0e60 ffff81022e4f0c00 ffff8102150d6140 >> ffff8102152ca1c0 ffff81022c7e6600 ffffffff885a4a8f ffff8102150d6140 >> ffffffff00000004 0000000000000032 ffff81022a338200 ffff81022a338af0 >> Call Trace: >> [] :svcrdma:svc_rdma_put_frmr+0xbc/0x117 >> [] :svcrdma:sq_cq_reap+0x11a/0x1a8 >> [] _spin_lock_bh+0x9/0x14 >> [] :svcrdma:dto_tasklet_func+0x13a/0x17a >> [] :mlx4_core:mlx4_eq_int+0x27e/0x28f >> [] tasklet_action+0x62/0xac >> [] __do_softirq+0x5e/0xd6 >> [] end_msi_irq_w_maskbit+0xf/0x1c >> [] call_softirq+0x1c/0x28 >> [] do_softirq+0x2c/0x85 >> [] do_IRQ+0xec/0xf5 >> [] mwait_idle+0x0/0x4a >> [] ret_from_intr+0x0/0xa >> [] mwait_idle+0x36/0x4a >> [] cpu_idle+0x95/0xb8 >> [] start_secondary+0x45a/0x469 >> >> >> Code: 49 8b 01 48 6b d2 38 48 83 e0 fc 48 01 d0 f0 0f ba 28 09 49 >> RIP [] mark_clean+0x50/0x77 >> RSP >> > > From tziporet at dev.mellanox.co.il Mon Mar 23 12:13:11 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Mon, 23 Mar 2009 12:13:11 -0700 Subject: [ofa-general][PATCH] NFS-RDMA fix for connectX fast registration wr, In-Reply-To: <49C7BEE6.7020907@mellanox.com> References: <49C422FC.7040303@mellanox.com> <49c6b719.0e538c0a.3e06.ffff8ed8@mx.google.com> <49c6ca6b.29578c0a.294a.ffffcd1a@mx.google.com> <49C7BEE6.7020907@mellanox.com> Message-ID: <49C7DF47.8030803@mellanox.co.il> Vu Pham wrote: > Tom, > >> Vu - I noticed this wasn't a usual kernel patch, so I've recoded it >> as a patch against Trond's tree, where I have the connectathon >> patches queued too. I can send it to him with your signed-off-by, >> if that's ok. >> >> > Here is the kernel patch. It's tested with latest connectX FW (2.6.636) > > >> I do want to be sure the patch is tested though. Does it work on >> earlier COnnectX firmware that may not support FRMR? I assume >> that fw doesn't set the bits the verbs.c file is looking for to detect >> FRMR support... correct? >> > > The patch does not work with old FW. > The old fw set the bit; however, it has bug on FRWR > > thanks, > -vu But I think we have to push the new code A drop of the FW with a fix is already available and it will be GA in few months Vu - do we need any fix for OFED 1.4.1 too? Tziporet From vuhuong at mellanox.com Mon Mar 23 12:42:43 2009 From: vuhuong at mellanox.com (Vu Pham) Date: Mon, 23 Mar 2009 12:42:43 -0700 Subject: [ofa-general][PATCH] NFS-RDMA fix for connectX fast registration wr, In-Reply-To: <49C7DF47.8030803@mellanox.co.il> References: <49C422FC.7040303@mellanox.com> <49c6b719.0e538c0a.3e06.ffff8ed8@mx.google.com> <49c6ca6b.29578c0a.294a.ffffcd1a@mx.google.com> <49C7BEE6.7020907@mellanox.com> <49C7DF47.8030803@mellanox.co.il> Message-ID: <49C7E633.1090003@mellanox.com> Tziporet Koren wrote: > Vu Pham wrote: >> Tom, >> >>> Vu - I noticed this wasn't a usual kernel patch, so I've recoded it >>> as a patch against Trond's tree, where I have the connectathon >>> patches queued too. I can send it to him with your signed-off-by, >>> if that's ok. >>> >>> >> Here is the kernel patch. It's tested with latest connectX FW (2.6.636) >> >> >>> I do want to be sure the patch is tested though. Does it work on >>> earlier COnnectX firmware that may not support FRMR? I assume >>> that fw doesn't set the bits the verbs.c file is looking for to detect >>> FRMR support... correct? >>> >> >> The patch does not work with old FW. >> The old fw set the bit; however, it has bug on FRWR >> >> thanks, >> -vu > > But I think we have to push the new code > A drop of the FW with a fix is already available and it will be GA in > few months > Vu - do we need any fix for OFED 1.4.1 too? Yes - I already sent to Tom patch to mainline kernel and to Jon the patch for ofed-1.4.1-rc2 -vu From weiny2 at llnl.gov Mon Mar 23 13:25:24 2009 From: weiny2 at llnl.gov (weiny2 at llnl.gov) Date: Mon, 23 Mar 2009 13:25:24 -0700 Subject: [ofa-general] OpenSM/IB management BOF at Sonoma In-Reply-To: <1237596444.22632.333.camel@auk31.llnl.gov> References: <20090320121358.GD6267@sashak.voltaire.com> <20090320100546.2f93de58.weiny2@llnl.gov> <20090321002532.GA8262@sashak.voltaire.com> <20090320172718.af93b06f.weiny2@llnl.gov> <1237596444.22632.333.camel@auk31.llnl.gov> Message-ID: <20090323132524.whpuvzp9uskkw8w8@www-openlabnet.llnl.gov> I signed us up for the "Palm Tree" room @5:00. The Palm Tree room is at the top of the stairs above the lobby. See you all there, Ira Quoting Al Chu : > >> Perhaps Tues night? I would be willing to skip some presentations >> perhaps even "OpenFabrics Alliance Annual Meeting" (Tues 5:30) to >> get started a bit earlier? > > Sounds good to me. > > Al > > On Fri, 2009-03-20 at 17:27 -0700, Ira Weiny wrote: >> If last year is any indication I think it will have to be after the >> presentations. >> >> Perhaps Tues night? I would be willing to skip some presentations >> perhaps even "OpenFabrics Alliance Annual Meeting" (Tues 5:30) to >> get started a bit earlier? >> >> Anyone else? >> >> Ira >> >> On Sat, 21 Mar 2009 02:25:32 +0200 >> Sasha Khapyorsky wrote: >> >> > On 10:05 Fri 20 Mar , Ira Weiny wrote: >> > > I will be there and yes it was successful. >> > >> > Good! Do you have any date/time preference? >> > >> > Sasha >> >> > -- > Albert Chu > chu11 at llnl.gov > Computer Scientist > High Performance Systems Division > Lawrence Livermore National Laboratory > > From tmtalpey at gmail.com Mon Mar 23 14:00:39 2009 From: tmtalpey at gmail.com (Tom Talpey) Date: Mon, 23 Mar 2009 17:00:39 -0400 Subject: ***SPAM*** Re: [ofa-general][PATCH] NFS-RDMA fix for connectX fast registration wr, In-Reply-To: <49C7DF47.8030803@mellanox.co.il> References: <49C422FC.7040303@mellanox.com> <49c6b719.0e538c0a.3e06.ffff8ed8@mx.google.com> <49c6ca6b.29578c0a.294a.ffffcd1a@mx.google.com> <49C7BEE6.7020907@mellanox.com> <49C7DF47.8030803@mellanox.co.il> Message-ID: <49c7f896.25578c0a.3652.627a@mx.google.com> At 03:13 PM 3/23/2009, Tziporet Koren wrote: >Vu Pham wrote: >> The patch does not work with old FW. >> The old fw set the bit; however, it has bug on FRWR >> > >But I think we have to push the new code >A drop of the FW with a fix is already available and it will be GA in >few months I don't fully understand - do you mean the need for setting the LOCAL_WRITE bit goes away with this upcoming ConnectX firmware? In other words, the NFS/RDMA is correct, currently, and Vu's patch is therefore purely a workaround for a firmware bug? I feel differently about pushing this fix upstream, if so. Tom. From jon at opengridcomputing.com Mon Mar 23 14:04:32 2009 From: jon at opengridcomputing.com (Jon Mason) Date: Mon, 23 Mar 2009 16:04:32 -0500 Subject: [ofa-general] Re: [PATCH] NFS-RDMA fix for connectX fast registration wr, In-Reply-To: <49C422FC.7040303@mellanox.com> References: <49C422FC.7040303@mellanox.com> Message-ID: <20090323210431.GA6861@opengridcomputing.com> On Fri, Mar 20, 2009 at 04:13:00PM -0700, Vu Pham wrote: > Hi Jon, > > Here is the patch for the file > kernel_patches/fixes/nfsrdma_to_2_6_28_18.patch. This patch will fix the > fast registration wr for Mellanox ConnectX. I was able to run Connectathon over Chelsio adapters without issue. Assuming everyone agrees on this solution, this patch should be safe for inclusion into the backports. Thanks, Jon > > ConnectX FRMR requires local write enable together with remote rdma > write enable > > thanks, > -vu > --- ofa_kernel-1.4.1.config/kernel_patches/fixes/nfsrdma_to_2.6.28_13.patch 2009-03-19 08:42:07.000000000 -0700 > +++ ofa_kernel-1.4.1/kernel_patches/fixes/nfsrdma_to_2.6.28_13.patch 2009-03-19 08:49:22.000000000 -0700 > @@ -167,7 +167,7 @@ > case RPCRDMA_MTHCAFMR: > case RPCRDMA_MEMWINDOWS_ASYNC: > case RPCRDMA_MEMWINDOWS: > -@@ -1391,6 +1457,96 @@ rpcrdma_unmap_one(struct rpcrdma_ia *ia, struct rpcrdma_mr_seg *seg) > +@@ -1391,6 +1457,98 @@ rpcrdma_unmap_one(struct rpcrdma_ia *ia, struct rpcrdma_mr_seg *seg) > } > > static int > @@ -214,8 +214,10 @@ > + frmr_wr.wr.fast_reg.page_list_len = i; > + frmr_wr.wr.fast_reg.page_shift = PAGE_SHIFT; > + frmr_wr.wr.fast_reg.length = i << PAGE_SHIFT; > -+ frmr_wr.wr.fast_reg.access_flags = (writing ? > -+ IB_ACCESS_REMOTE_WRITE : IB_ACCESS_REMOTE_READ); > ++ frmr_wr.wr.fast_reg.access_flags = > ++ (writing ? > ++ (IB_ACCESS_REMOTE_WRITE | IB_ACCESS_LOCAL_WRITE) : > ++ IB_ACCESS_REMOTE_READ); > + frmr_wr.wr.fast_reg.rkey = seg1->mr_chunk.rl_mw->r.frmr.fr_mr->rkey; > + DECR_CQCOUNT(&r_xprt->rx_ep); > + From vuhuong at mellanox.com Mon Mar 23 14:14:45 2009 From: vuhuong at mellanox.com (Vu Pham) Date: Mon, 23 Mar 2009 14:14:45 -0700 Subject: [ofa-general][PATCH] NFS-RDMA fix for connectX fast registration wr, In-Reply-To: <49c7f896.25578c0a.3652.627a@mx.google.com> References: <49C422FC.7040303@mellanox.com> <49c6b719.0e538c0a.3e06.ffff8ed8@mx.google.com> <49c6ca6b.29578c0a.294a.ffffcd1a@mx.google.com> <49C7BEE6.7020907@mellanox.com> <49C7DF47.8030803@mellanox.co.il> <49c7f896.25578c0a.3652.627a@mx.google.com> Message-ID: <49C7FBC5.6020902@mellanox.com> Tom Talpey wrote: > At 03:13 PM 3/23/2009, Tziporet Koren wrote: > >> Vu Pham wrote: >> >>> The patch does not work with old FW. >>> The old fw set the bit; however, it has bug on FRWR >>> >>> >> But I think we have to push the new code >> A drop of the FW with a fix is already available and it will be GA in >> few months >> > > I don't fully understand - do you mean the need for setting the > LOCAL_WRITE bit goes away with this upcoming ConnectX firmware? > No - the setting for LOCAL_WRITE bit will still be needed for this upcoming ConnectX FW. > In other words, the NFS/RDMA is correct, currently, and Vu's patch > is therefore purely a workaround for a firmware bug? > > I feel differently about pushing this fix upstream, if so. > > Tom. > > From tmtalpey at gmail.com Mon Mar 23 14:18:35 2009 From: tmtalpey at gmail.com (Tom Talpey) Date: Mon, 23 Mar 2009 17:18:35 -0400 Subject: ***SPAM*** Re: [ofa-general][PATCH] NFS-RDMA fix for connectX fast registration wr, In-Reply-To: <49C7FBC5.6020902@mellanox.com> References: <49C422FC.7040303@mellanox.com> <49c6b719.0e538c0a.3e06.ffff8ed8@mx.google.com> <49c6ca6b.29578c0a.294a.ffffcd1a@mx.google.com> <49C7BEE6.7020907@mellanox.com> <49C7DF47.8030803@mellanox.co.il> <49c7f896.25578c0a.3652.627a@mx.google.com> <49C7FBC5.6020902@mellanox.com> Message-ID: <49c7fcb1.1cbc720a.0e30.ffffaf1d@mx.google.com> At 05:14 PM 3/23/2009, Vu Pham wrote: >Tom Talpey wrote: >> At 03:13 PM 3/23/2009, Tziporet Koren wrote: >> >>> Vu Pham wrote: >>> >>>> The patch does not work with old FW. >>>> The old fw set the bit; however, it has bug on FRWR >>>> >>>> >>> But I think we have to push the new code >>> A drop of the FW with a fix is already available and it will be GA in >>> few months >>> >> >> I don't fully understand - do you mean the need for setting the >> LOCAL_WRITE bit goes away with this upcoming ConnectX firmware? >> > >No - the setting for LOCAL_WRITE bit will still be needed for this >upcoming ConnectX FW. Ok, two question in that case... 1) What's the firmware fix that Tziporet mentions actually for? 2) Is the need for LOCAL_WRITE documented at the FRMR verb layer, and do other implementations agree that it's harmless? Tom. > > >> In other words, the NFS/RDMA is correct, currently, and Vu's patch >> is therefore purely a workaround for a firmware bug? >> >> I feel differently about pushing this fix upstream, if so. >> >> Tom. >> >> > > From tmtalpey at gmail.com Mon Mar 23 14:24:50 2009 From: tmtalpey at gmail.com (Tom Talpey) Date: Mon, 23 Mar 2009 17:24:50 -0400 Subject: [ofa-general] ***SPAM*** All gmail/yahoo messages flagged as s-p-a-m? Message-ID: <49c7fe41.2a528c0a.296c.4b81@mx.google.com> It appears that all posters from gmail and yahoo, at least, have their messages flagged by the openfabrics.org server as spam. Is this intentional? Tom. From vuhuong at mellanox.com Mon Mar 23 14:29:17 2009 From: vuhuong at mellanox.com (Vu Pham) Date: Mon, 23 Mar 2009 14:29:17 -0700 Subject: [ofa-general][PATCH] NFS-RDMA fix for connectX fast registration wr, In-Reply-To: <49c7fcb1.1cbc720a.0e30.ffffaf1d@mx.google.com> References: <49C422FC.7040303@mellanox.com> <49c6b719.0e538c0a.3e06.ffff8ed8@mx.google.com> <49c6ca6b.29578c0a.294a.ffffcd1a@mx.google.com> <49C7BEE6.7020907@mellanox.com> <49C7DF47.8030803@mellanox.co.il> <49c7f896.25578c0a.3652.627a@mx.google.com> <49C7FBC5.6020902@mellanox.com> <49c7fcb1.1cbc720a.0e30.ffffaf1d@mx.google.com> Message-ID: <49C7FF2D.3090309@mellanox.com> Tom Talpey wrote: > At 05:14 PM 3/23/2009, Vu Pham wrote: > >> Tom Talpey wrote: >> >>> At 03:13 PM 3/23/2009, Tziporet Koren wrote: >>> >>> >>>> Vu Pham wrote: >>>> >>>> >>>>> The patch does not work with old FW. >>>>> The old fw set the bit; however, it has bug on FRWR >>>>> >>>>> >>>>> >>>> But I think we have to push the new code >>>> A drop of the FW with a fix is already available and it will be GA in >>>> few months >>>> >>>> >>> I don't fully understand - do you mean the need for setting the >>> LOCAL_WRITE bit goes away with this upcoming ConnectX firmware? >>> >>> >> No - the setting for LOCAL_WRITE bit will still be needed for this >> upcoming ConnectX FW. >> > > Ok, two question in that case... > > 1) What's the firmware fix that Tziporet mentions actually for? > A FW bug on Fast registration WR operation > 2) Is the need for LOCAL_WRITE documented at the FRMR verb layer, > and do other implementations agree that it's harmless? > > It is documented in ConnectX's PRM (Program Reference Manual) Jon already verify the patch and it's harmless to Chelsio. -vu > Tom. > > >> >>> In other words, the NFS/RDMA is correct, currently, and Vu's patch >>> is therefore purely a workaround for a firmware bug? >>> >>> I feel differently about pushing this fix upstream, if so. >>> >>> Tom. >>> >>> >>> >> > > From sashak at voltaire.com Mon Mar 23 15:17:07 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Mar 2009 00:17:07 +0200 Subject: [ofa-general] Re: [PATCH] opensm/include/ib_types.h: Fix some typos In-Reply-To: <20090323122820.GA19210@comcast.net> References: <20090323122820.GA19210@comcast.net> Message-ID: <20090323221707.GA31148@sashak.voltaire.com> On 07:28 Mon 23 Mar , Hal Rosenstock wrote: > > Also, some cosmetic formatting changes > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Mon Mar 23 15:17:45 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Mar 2009 00:17:45 +0200 Subject: [ofa-general] Re: [PATCH] opensm: Some cosmetic changes In-Reply-To: <20090323122922.GB19210@comcast.net> References: <20090323122922.GB19210@comcast.net> Message-ID: <20090323221745.GB31148@sashak.voltaire.com> On 07:29 Mon 23 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Mon Mar 23 15:39:06 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Mar 2009 00:39:06 +0200 Subject: [ofa-general] Re: [PATCH] opensm: Remove __osm_ prefixes In-Reply-To: <20090323123012.GC19210@comcast.net> References: <20090323123012.GC19210@comcast.net> Message-ID: <20090323223853.GC31148@sashak.voltaire.com> On 07:30 Mon 23 Mar , Hal Rosenstock wrote: > > Also, some cosmetic formatting changes Applied. Thanks. Sasha From sashak at voltaire.com Mon Mar 23 15:40:44 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Mar 2009 00:40:44 +0200 Subject: [ofa-general] Re: [PATCH] libibmad/sa.c: Cosmetic formatting changes In-Reply-To: <20090321120840.GA10982@comcast.net> References: <20090321120840.GA10982@comcast.net> Message-ID: <20090323224044.GD31148@sashak.voltaire.com> On 07:08 Sat 21 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Mon Mar 23 18:12:19 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Mar 2009 03:12:19 +0200 Subject: [ofa-general] Re: [PATCH] ib-diag/vendstat: counter-group-info & config-counter-group vendor mads In-Reply-To: References: Message-ID: <20090324011217.GH31148@sashak.voltaire.com> On 14:27 Thu 19 Mar , Or Gerlitz wrote: > Implement counter-group-info and config-counter-group IS4 vendor mads. > The config-counter-group mad should be used to configure a switch port > to make counters such as PortXmitDataSL or PortRcvDataSL active. > > Signed-off-by: Or Gerlitz Applied. We are waiting now for vendstat man page update patch. Thanks. Sasha From sashak at voltaire.com Mon Mar 23 18:50:56 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Mar 2009 03:50:56 +0200 Subject: [ofa-general] Re: [PATCHv2] libibmad: Add PortSelect and CounterSelect fields for PortXmit/RcvDataSL In-Reply-To: <20090320231101.GA1424@comcast.net> References: <20090320231101.GA1424@comcast.net> Message-ID: <20090324015056.GJ31148@sashak.voltaire.com> On 18:11 Fri 20 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Apllied. Thanks. Sasha From sashak at voltaire.com Mon Mar 23 18:59:14 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Mar 2009 03:59:14 +0200 Subject: [ofa-general] Couple of questions about OpenSM In-Reply-To: <49C49E9D.6080209@morey-chaisemartin.com> References: <49C21A7F.3090203@ext.bull.net> <20090320115941.GC6267@sashak.voltaire.com> <49C3BE63.1060703@morey-chaisemartin.com> <20090321013805.GE8262@sashak.voltaire.com> <49C49C8F.8000608@morey-chaisemartin.com> <49C49E9D.6080209@morey-chaisemartin.com> Message-ID: <20090324015914.GK31148@sashak.voltaire.com> On 09:00 Sat 21 Mar , Nicolas Morey-Chaisemartin wrote: > > Fat-Tree needs the hops table to be set even for CA lics to detect if we are coming from a shorter path or not... But is it really needed? Hops to CA lid is always hops to its neighbor switch plus one, no? Sasha From devel at morey-chaisemartin.com Mon Mar 23 23:16:44 2009 From: devel at morey-chaisemartin.com (Nicolas Morey-Chaisemartin) Date: Tue, 24 Mar 2009 07:16:44 +0100 Subject: [ofa-general] Couple of questions about OpenSM In-Reply-To: <20090324015914.GK31148@sashak.voltaire.com> References: <49C21A7F.3090203@ext.bull.net> <20090320115941.GC6267@sashak.voltaire.com> <49C3BE63.1060703@morey-chaisemartin.com> <20090321013805.GE8262@sashak.voltaire.com> <49C49C8F.8000608@morey-chaisemartin.com> <49C49E9D.6080209@morey-chaisemartin.com> <20090324015914.GK31148@sashak.voltaire.com> Message-ID: <49C87ACC.9000002@morey-chaisemartin.com> Yes it is necessary because we use this thing to see if we have already created routes toward a node or not. If we use the sw lid we can't know that for sure, except if we merge all the routes to CA under the same leaf switch. And this is not a good behaviour! However I fixed the problem in my patch. We use our own minhop tables which is much smaller but we still update the osm_sw hop table if target lid is a switch. Sasha Khapyorsky a écrit : > On 09:00 Sat 21 Mar , Nicolas Morey-Chaisemartin wrote: >> Fat-Tree needs the hops table to be set even for CA lics to detect if we are coming from a shorter path or not... > > But is it really needed? Hops to CA lid is always hops to its neighbor > switch plus one, no? > > Sasha > > From vlad at lists.openfabrics.org Tue Mar 24 03:22:49 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Tue, 24 Mar 2009 03:22:49 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090324-0200 daily build status Message-ID: <20090324102249.E4CD0E60E43@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From hnrose at comcast.net Tue Mar 24 05:16:42 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 24 Mar 2009 07:16:42 -0500 Subject: [ofa-general] ***SPAM*** [PATCH][TRIVIAL] libibmad/dump.c: Cosmetic formatting changes Message-ID: <20090324121642.GA23935@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/libibmad/src/dump.c b/libibmad/src/dump.c index 4f71615..051c708 100644 --- a/libibmad/src/dump.c +++ b/libibmad/src/dump.c @@ -703,16 +703,20 @@ void mad_dump_perfcounters_xmt_sl(char *buf, int bufsz, void *val, int valsz) { int cnt; - cnt = _dump_fields(buf, bufsz, val, IB_PC_EXT_PORT_SELECT_F, IB_PC_EXT_XMT_BYTES_F); - _dump_fields(buf + cnt, bufsz - cnt, val, IB_PC_XMT_DATA_SL_FIRST_F, IB_PC_XMT_DATA_SL_LAST_F); + cnt = _dump_fields(buf, bufsz, val, IB_PC_EXT_PORT_SELECT_F, + IB_PC_EXT_XMT_BYTES_F); + _dump_fields(buf + cnt, bufsz - cnt, val, IB_PC_XMT_DATA_SL_FIRST_F, + IB_PC_XMT_DATA_SL_LAST_F); } void mad_dump_perfcounters_rcv_sl(char *buf, int bufsz, void *val, int valsz) { int cnt; - cnt = _dump_fields(buf, bufsz, val, IB_PC_EXT_PORT_SELECT_F, IB_PC_EXT_XMT_BYTES_F); - _dump_fields(buf + cnt, bufsz - cnt, val, IB_PC_RCV_DATA_SL_FIRST_F, IB_PC_RCV_DATA_SL_LAST_F); + cnt = _dump_fields(buf, bufsz, val, IB_PC_EXT_PORT_SELECT_F, + IB_PC_EXT_XMT_BYTES_F); + _dump_fields(buf + cnt, bufsz - cnt, val, IB_PC_RCV_DATA_SL_FIRST_F, + IB_PC_RCV_DATA_SL_LAST_F); } void xdump(FILE * file, char *msg, void *p, int size) @@ -737,7 +741,6 @@ void xdump(FILE * file, char *msg, void *p, int size) fputc('\n', file); cp += 2; } - if (i % 16) { + if (i % 16) fputc('\n', file); - } } From hnrose at comcast.net Tue Mar 24 05:10:25 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 24 Mar 2009 07:10:25 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] opensm/iba/ib_types.h: Add PortXmit/RcvDataSL PerfMgt attributes Message-ID: <20090324121025.GA23604@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/include/iba/ib_types.h b/opensm/include/iba/ib_types.h index 32b916e..e148597 100644 --- a/opensm/include/iba/ib_types.h +++ b/opensm/include/iba/ib_types.h @@ -1042,6 +1042,30 @@ static inline boolean_t OSM_API ib_class_is_rmpp(IN const uint8_t class_code) #define IB_MAD_ATTR_PORT_CNTRS_EXT (CL_HTON16(0x001D)) /**********/ +/****d* IBA Base: Constants/IB_MAD_ATTR_PORT_XMIT_DATA_SL +* NAME +* IB_MAD_ATTR_PORT_XMIT_DATA_SL +* +* DESCRIPTION +* PortXmitDataSL attribute (A13.6.4) +* +* SOURCE +*/ +#define IB_MAD_ATTR_PORT_XMIT_DATA_SL (CL_HTON16(0x0036)) +/**********/ + +/****d* IBA Base: Constants/IB_MAD_ATTR_PORT_RCV_DATA_SL +* NAME +* IB_MAD_ATTR_PORT_RCV_DATA_SL +* +* DESCRIPTION +* PortRcvDataSL attribute (A13.6.4) +* +* SOURCE +*/ +#define IB_MAD_ATTR_PORT_RCV_DATA_SL (CL_HTON16(0x0037)) +/**********/ + /****d* IBA Base: Constants/IB_MAD_ATTR_GUID_INFO * NAME * IB_MAD_ATTR_GUID_INFO @@ -7812,6 +7836,44 @@ typedef struct _ib_port_samples_result { } PACK_SUFFIX ib_port_samples_result_t; #include +/****s* IBA Base: Types/ib_port_xmit_data_sl +* NAME +* ib_port_xmit_data_sl_t +* +* DESCRIPTION +* IBA defined PortXmitDataSL Attribute. (A13.6.4) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_port_xmit_data_sl { + uint8_t reserved; + uint8_t port_select; + ib_net16_t counter_select; + ib_net32_t port_xmit_data_sl[16]; + uint8_t resv[124]; +} PACK_SUFFIX ib_port_xmit_data_sl_t; +#include + +/****s* IBA Base: Types/ib_port_rcv_data_sl +* NAME +* ib_port_rcv_data_sl_t +* +* DESCRIPTION +* IBA defined PortRcvDataSL Attribute. (A13.6.4) +* +* SYNOPSIS +*/ +#include +typedef struct _ib_port_rcv_data_sl { + uint8_t reserved; + uint8_t port_select; + ib_net16_t counter_select; + ib_net32_t port_rcv_data_sl[16]; + uint8_t resv[124]; +} PACK_SUFFIX ib_port_rcv_data_sl_t; +#include + /****d* IBA Base: Types/DM_SVC_NAME * NAME * DM_SVC_NAME From hnrose at comcast.net Tue Mar 24 09:15:01 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 24 Mar 2009 11:15:01 -0500 Subject: [ofa-general] ***SPAM*** [PATCH][TRIVIAL] opensm/osm_sm_state_mgr.c: Remove unneeded return statement Message-ID: <20090324161501.GB17598@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_sm_state_mgr.c b/opensm/opensm/osm_sm_state_mgr.c index 9a402be..d06e6aa 100644 --- a/opensm/opensm/osm_sm_state_mgr.c +++ b/opensm/opensm/osm_sm_state_mgr.c @@ -217,7 +217,6 @@ void osm_sm_state_mgr_polling_callback(IN void *context) Exit: OSM_LOG_EXIT(sm->p_log); - return; } /********************************************************************** From hnrose at comcast.net Tue Mar 24 09:14:13 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 24 Mar 2009 11:14:13 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] opensm/osm_sminfo_rcv.c: Minor simplification Message-ID: <20090324161413.GA17598@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_sminfo_rcv.c b/opensm/opensm/osm_sminfo_rcv.c index d3f6678..dd2000f 100644 --- a/opensm/opensm/osm_sminfo_rcv.c +++ b/opensm/opensm/osm_sminfo_rcv.c @@ -334,8 +334,7 @@ static osm_signal_t smi_rcv_process_get_sm(IN osm_sm_t * sm, break; case IB_SMINFO_STATE_DISCOVERING: case IB_SMINFO_STATE_STANDBY: - if (smi_rcv_remote_sm_is_higher(sm, p_smi) - == TRUE) { + if (smi_rcv_remote_sm_is_higher(sm, p_smi)) { /* the remote is a higher sm - need to stop sweeping */ sm->master_sm_found = 1; /* save on the sm the guid of the higher SM we found - */ From sean.hefty at intel.com Tue Mar 24 09:14:13 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 24 Mar 2009 09:14:13 -0700 Subject: [ofa-general] RE: [ofw] What is the current support level for QoS in WinOF? In-Reply-To: <008801c9ac52$f3abad20$05c8a8c0@DIEGO> References: <018301c9aba1$0a0c4370$05c8a8c0@DIEGO> <18C53B9CBFF14CDB930F6E52B77BD9D0@amr.corp.intel.com> <008801c9ac52$f3abad20$05c8a8c0@DIEGO> Message-ID: >I do have urgent time-sensitive traffic, and non-urgent traffic. >The urgent traffic and the non-urgent traffic is generated from different >hosts. >I would like to differentiate them by using different SL, then configure QoS to >give maximum priority to the urgent time-sensitive >traffic, and minimum priority to the non-urgent traffic. > >Are you saying I can't do this in WinOF? I don't think the WinOF opensm will support this, but I'm not certain. >And I can't do that even adding a Linux host that runs opensm (OFED version)? I would expect that this is possible. >Traffic separation based on HCA port could be an option, but I need to think >more about that. >What can you do with that kind of QoS? More simply, this would allow you to group hosts into different traffic priority groups. >Do you mark this HCA port as high-priority, that HCA port as low-priority, etc? >What happens when a high-pri port sends traffic to a low-pri port? And vice- >versa? There should be rules in the opensm QoS config file that will determine this. I've copied the general list on this reply. Sasha, Hal, or someone that deals more directly with opensm will be able to direct you better. >What happens when a high-pri port sends traffic to a "normal" port (a port that >is not marked as high-priority nor low-priority)? >I'm using only RDMA Write with Imm in my system, although I'm interested in >what happens on all types of traffic. >If you know of a document that explains that, please let me know, I haven't >found it by now. The OFED opensm includes documentation on setting up QoS. It's in opensm/doc in the management source tree. - Sean From hal.rosenstock at gmail.com Tue Mar 24 09:46:55 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 24 Mar 2009 12:46:55 -0400 Subject: ***SPAM*** Re: [ofa-general] RE: [ofw] What is the current support level for QoS in WinOF? In-Reply-To: References: <018301c9aba1$0a0c4370$05c8a8c0@DIEGO> <18C53B9CBFF14CDB930F6E52B77BD9D0@amr.corp.intel.com> <008801c9ac52$f3abad20$05c8a8c0@DIEGO> Message-ID: On Tue, Mar 24, 2009 at 12:14 PM, Sean Hefty wrote: >>I do have urgent time-sensitive traffic, and non-urgent traffic. >>The urgent traffic and the non-urgent traffic is generated from different >>hosts. >>I would like to differentiate them by using different SL, then configure QoS to >>give maximum priority to the urgent time-sensitive >>traffic, and minimum priority to the non-urgent traffic. >> >>Are you saying I can't do this in WinOF? > > I don't think the WinOF opensm will support this, but I'm not certain. Primitive QoS has some support for this but not sure it will be complete enough or integrated enough with the end stack for your purposes depending on how the SL used by the application is to be set. >>And I can't do that even adding a Linux host that runs opensm (OFED version)? > > I would expect that this is possible. > >>Traffic separation based on HCA port could be an option, but I need to think >>more about that. >>What can you do with that kind of QoS? > > More simply, this would allow you to group hosts into different traffic priority > groups. > >>Do you mark this HCA port as high-priority, that HCA port as low-priority, etc? >>What happens when a high-pri port sends traffic to a low-pri port? And vice- >>versa? > > There should be rules in the opensm QoS config file that will determine this. > I've copied the general list on this reply.  Sasha, Hal, or someone that deals > more directly with opensm will be able to direct you better. > >>What happens when a high-pri port sends traffic to a "normal" port (a port that >>is not marked as high-priority nor low-priority)? >>I'm using only RDMA Write with Imm in my system, although I'm interested in >>what happens on all types of traffic. >>If you know of a document that explains that, please let me know, I haven't >>found it by now. > > The OFED opensm includes documentation on setting up QoS.  It's in opensm/doc in > the management source tree. There are 2 docs: qos-config.txt (describes "primitive QoS support") and QoS_management_in_OpenSM.txt (describes QoS Annex support). -- Hal > - Sean > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hnrose at comcast.net Tue Mar 24 11:25:10 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 24 Mar 2009 13:25:10 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] ibsim: Add SMSL support to PortInfo attribute Message-ID: <20090324182510.GA18072@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/ibsim/sim.h b/ibsim/sim.h index 5e9b4f0..4c1439c 100644 --- a/ibsim/sim.h +++ b/ibsim/sim.h @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This file is part of ibsim. * @@ -189,6 +190,7 @@ struct Port { int portnum; int lid; int smlid; + int smsl; int linkwidth; int linkwidthena; int linkspeed; diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c index d2c313c..a24e5d3 100644 --- a/ibsim/sim_mad.c +++ b/ibsim/sim_mad.c @@ -432,6 +432,7 @@ do_portinfo(Port * port, unsigned op, uint32_t portnum, uint8_t * data) } p->lid = newlid; p->smlid = mad_get_field(data, 0, IB_PORT_SMLID_F); + p->smsl = mad_get_field(data, 0, IB_PORT_SMSL_F); // p->linkwidth = mad_get_field(data, 0, IB_PORT_LINK_WIDTH_ENABLED_F); // ignored p->lmc = mad_get_field(data, 0, IB_PORT_LMC_F); p->hoqlife = mad_get_field(data, 0, IB_PORT_HOQ_LIFE_F); diff --git a/ibsim/sim_net.c b/ibsim/sim_net.c index 13c3b8c..8b95947 100644 --- a/ibsim/sim_net.c +++ b/ibsim/sim_net.c @@ -1,5 +1,6 @@ /* * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. + * Copyright (c) 2009 HNR Consulting. All rights reserved. * * This file is part of ibsim. * @@ -1077,6 +1078,7 @@ void update_portinfo(Port * p) p->node->type == SWITCH_NODE ? 0 : p->portnum); mad_set_field(pi, 0, IB_PORT_LID_F, p->lid); mad_set_field(pi, 0, IB_PORT_SMLID_F, p->smlid); + mad_set_field(pi, 0, IB_PORT_SMSL_F, p->smsl); mad_set_field(pi, 0, IB_PORT_OPER_VLS_F, p->op_vls); mad_set_field(pi, 0, IB_PORT_LINK_WIDTH_ENABLED_F, p->linkwidthena); mad_set_field(pi, 0, IB_PORT_LINK_WIDTH_SUPPORTED_F, From hnrose at comcast.net Tue Mar 24 11:44:48 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 24 Mar 2009 13:44:48 -0500 Subject: [ofa-general] [PATCH][TRIVIAL] opensm: Remove some __ prefixes Message-ID: <20090324184448.GA18569@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_prtn.c b/opensm/opensm/osm_prtn.c index 656d011..3f4cf00 100644 --- a/opensm/opensm/osm_prtn.c +++ b/opensm/opensm/osm_prtn.c @@ -251,7 +251,7 @@ ib_api_status_t osm_prtn_add_mcgroup(osm_log_t * p_log, return status; } -static uint16_t __generate_pkey(osm_subn_t * p_subn) +static uint16_t generate_pkey(osm_subn_t * p_subn) { uint16_t pkey; @@ -291,7 +291,7 @@ osm_prtn_t *osm_prtn_make_new(osm_log_t * p_log, osm_subn_t * p_subn, if (!pkey) { if (name && (p = osm_prtn_find_by_name(p_subn, name))) return p; - if (!(pkey = __generate_pkey(p_subn))) + if (!(pkey = generate_pkey(p_subn))) return NULL; } diff --git a/opensm/opensm/osm_sa_class_port_info.c b/opensm/opensm/osm_sa_class_port_info.c index 6d5ad0b..d2ab96a 100644 --- a/opensm/opensm/osm_sa_class_port_info.c +++ b/opensm/opensm/osm_sa_class_port_info.c @@ -57,7 +57,7 @@ #define MAX_MSECS_TO_RTV 24 /* Precalculated table in msec (index is related to encoded value) */ /* 4.096 usec * 2 ** n (where n = 8 - 31) */ -const static uint32_t __msecs_to_rtv_table[MAX_MSECS_TO_RTV] = { +const static uint32_t msecs_to_rtv_table[MAX_MSECS_TO_RTV] = { 1, 2, 4, 8, 16, 33, 67, 134, 268, 536, 1073, 2147, @@ -109,12 +109,12 @@ static void cpi_rcv_respond(IN osm_sa_t * sa, IN const osm_madw_t * p_madw) /* Calculate encoded response time value */ /* transaction timeout is in msec */ if (sa->p_subn->opt.transaction_timeout > - __msecs_to_rtv_table[MAX_MSECS_TO_RTV - 1]) + msecs_to_rtv_table[MAX_MSECS_TO_RTV - 1]) rtv = MAX_MSECS_TO_RTV - 1; else { for (rtv = 0; rtv < MAX_MSECS_TO_RTV; rtv++) { if (sa->p_subn->opt.transaction_timeout <= - __msecs_to_rtv_table[rtv]) + msecs_to_rtv_table[rtv]) break; } } diff --git a/opensm/opensm/osm_service.c b/opensm/opensm/osm_service.c index b7c1270..cc834b2 100644 --- a/opensm/opensm/osm_service.c +++ b/opensm/opensm/osm_service.c @@ -92,8 +92,8 @@ osm_svcr_t *osm_svcr_new(IN const ib_service_record_t * p_svc_rec) **********************************************************************/ static cl_status_t -__match_rid_of_svc_rec(IN const cl_list_item_t * const p_list_item, - IN void *context) +match_rid_of_svc_rec(IN const cl_list_item_t * const p_list_item, + IN void *context) { ib_service_record_t *p_svc_rec = (ib_service_record_t *) context; osm_svcr_t *p_svcr = (osm_svcr_t *) p_list_item; @@ -123,7 +123,7 @@ osm_svcr_t *osm_svcr_get_by_rid(IN osm_subn_t const *p_subn, OSM_LOG_ENTER(p_log); p_list_item = cl_qlist_find_from_head(&p_subn->sa_sr_list, - __match_rid_of_svc_rec, + match_rid_of_svc_rec, p_svc_rec); if (p_list_item == cl_qlist_end(&p_subn->sa_sr_list)) diff --git a/opensm/opensm/osm_ucast_cache.c b/opensm/opensm/osm_ucast_cache.c index c89e24d..216b496 100644 --- a/opensm/opensm/osm_ucast_cache.c +++ b/opensm/opensm/osm_ucast_cache.c @@ -80,7 +80,7 @@ typedef struct cache_switch { /********************************************************************** **********************************************************************/ -static uint16_t __cache_sw_get_base_lid_ho(cache_switch_t * p_sw) +static uint16_t cache_sw_get_base_lid_ho(cache_switch_t * p_sw) { return p_sw->ports[0].remote_lid_ho; } @@ -88,7 +88,7 @@ static uint16_t __cache_sw_get_base_lid_ho(cache_switch_t * p_sw) /********************************************************************** **********************************************************************/ -static boolean_t __cache_sw_is_leaf(cache_switch_t * p_sw) +static boolean_t cache_sw_is_leaf(cache_switch_t * p_sw) { return p_sw->ports[0].is_leaf; } @@ -96,7 +96,7 @@ static boolean_t __cache_sw_is_leaf(cache_switch_t * p_sw) /********************************************************************** **********************************************************************/ -static void __cache_sw_set_leaf(cache_switch_t * p_sw) +static void cache_sw_set_leaf(cache_switch_t * p_sw) { p_sw->ports[0].is_leaf = TRUE; } @@ -104,7 +104,7 @@ static void __cache_sw_set_leaf(cache_switch_t * p_sw) /********************************************************************** **********************************************************************/ -static cache_switch_t *__cache_sw_new(uint16_t lid_ho, unsigned num_ports) +static cache_switch_t *cache_sw_new(uint16_t lid_ho, unsigned num_ports) { cache_switch_t *p_cache_sw = malloc(sizeof(cache_switch_t) + num_ports * sizeof(cache_port_t)); @@ -126,7 +126,7 @@ static cache_switch_t *__cache_sw_new(uint16_t lid_ho, unsigned num_ports) /********************************************************************** **********************************************************************/ -static void __cache_sw_destroy(cache_switch_t * p_sw) +static void cache_sw_destroy(cache_switch_t * p_sw) { if (!p_sw) return; @@ -141,7 +141,7 @@ static void __cache_sw_destroy(cache_switch_t * p_sw) /********************************************************************** **********************************************************************/ -static cache_switch_t *__cache_get_sw(osm_ucast_mgr_t * p_mgr, uint16_t lid_ho) +static cache_switch_t *cache_get_sw(osm_ucast_mgr_t * p_mgr, uint16_t lid_ho) { cache_switch_t *p_cache_sw = (cache_switch_t *) cl_qmap_get(&p_mgr->cache_sw_tbl, lid_ho); @@ -154,8 +154,8 @@ static cache_switch_t *__cache_get_sw(osm_ucast_mgr_t * p_mgr, uint16_t lid_ho) /********************************************************************** **********************************************************************/ -static void __cache_add_sw_link(osm_ucast_mgr_t * p_mgr, osm_physp_t *p, - uint16_t remote_lid_ho, boolean_t is_ca) +static void cache_add_sw_link(osm_ucast_mgr_t * p_mgr, osm_physp_t *p, + uint16_t remote_lid_ho, boolean_t is_ca) { cache_switch_t *p_cache_sw; uint16_t lid_ho = cl_ntoh16(osm_node_get_base_lid(p->p_node, 0)); @@ -169,9 +169,9 @@ static void __cache_add_sw_link(osm_ucast_mgr_t * p_mgr, osm_physp_t *p, "Caching switch port: lid %u [port %u] -> lid %u (%s)\n", lid_ho, p->port_num, remote_lid_ho, (is_ca) ? "CA/RTR" : "SW"); - p_cache_sw = __cache_get_sw(p_mgr, lid_ho); + p_cache_sw = cache_get_sw(p_mgr, lid_ho); if (!p_cache_sw) { - p_cache_sw = __cache_sw_new(lid_ho, p->p_node->sw->num_ports); + p_cache_sw = cache_sw_new(lid_ho, p->p_node->sw->num_ports); if (!p_cache_sw) { OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR AD01: Out of memory - cache is invalid\n"); @@ -190,7 +190,7 @@ static void __cache_add_sw_link(osm_ucast_mgr_t * p_mgr, osm_physp_t *p, } if (is_ca) - __cache_sw_set_leaf(p_cache_sw); + cache_sw_set_leaf(p_cache_sw); if (p_cache_sw->ports[p->port_num].remote_lid_ho == 0) { /* cache this link only if it hasn't been already cached */ @@ -204,7 +204,7 @@ Exit: /********************************************************************** **********************************************************************/ -static void __cache_cleanup_switches(osm_ucast_mgr_t * p_mgr) +static void cache_cleanup_switches(osm_ucast_mgr_t * p_mgr) { cache_switch_t *p_sw; cache_switch_t *p_next_sw; @@ -228,7 +228,7 @@ static void __cache_cleanup_switches(osm_ucast_mgr_t * p_mgr) if (!found_port) { cl_qmap_remove_item(&p_mgr->cache_sw_tbl, &p_sw->map_item); - __cache_sw_destroy(p_sw); + cache_sw_destroy(p_sw); } } } @@ -237,8 +237,8 @@ static void __cache_cleanup_switches(osm_ucast_mgr_t * p_mgr) **********************************************************************/ static void -__cache_check_link_change(osm_ucast_mgr_t * p_mgr, - osm_physp_t * p_physp_1, osm_physp_t * p_physp_2) +cache_check_link_change(osm_ucast_mgr_t * p_mgr, + osm_physp_t * p_physp_1, osm_physp_t * p_physp_2) { OSM_LOG_ENTER(p_mgr->p_log); CL_ASSERT(p_physp_1 && p_physp_2); @@ -268,9 +268,9 @@ Exit: /********************************************************************** **********************************************************************/ -static void __cache_remove_port(osm_ucast_mgr_t * p_mgr, uint16_t lid_ho, - uint8_t port_num, uint16_t remote_lid_ho, - boolean_t is_ca) +static void cache_remove_port(osm_ucast_mgr_t * p_mgr, uint16_t lid_ho, + uint8_t port_num, uint16_t remote_lid_ho, + boolean_t is_ca) { cache_switch_t *p_cache_sw; @@ -279,7 +279,7 @@ static void __cache_remove_port(osm_ucast_mgr_t * p_mgr, uint16_t lid_ho, if (!p_mgr->cache_valid) goto Exit; - p_cache_sw = __cache_get_sw(p_mgr, lid_ho); + p_cache_sw = cache_get_sw(p_mgr, lid_ho); if (!p_cache_sw) { OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, "Found uncached switch/link (lid %u, port %u) - " @@ -327,14 +327,14 @@ static void __cache_remove_port(osm_ucast_mgr_t * p_mgr, uint16_t lid_ho, p_cache_sw->ports[port_num].is_leaf = FALSE; Exit: OSM_LOG_EXIT(p_mgr->p_log); -} /* __cache_remove_port() */ +} /* cache_remove_port() */ /********************************************************************** **********************************************************************/ static void -__cache_restore_ucast_info(osm_ucast_mgr_t * p_mgr, - cache_switch_t * p_cache_sw, osm_switch_t * p_sw) +cache_restore_ucast_info(osm_ucast_mgr_t * p_mgr, + cache_switch_t * p_cache_sw, osm_switch_t * p_sw) { if (!p_mgr->cache_valid) return; @@ -362,7 +362,7 @@ __cache_restore_ucast_info(osm_ucast_mgr_t * p_mgr, /********************************************************************** **********************************************************************/ -static void __ucast_cache_dump(osm_ucast_mgr_t * p_mgr) +static void ucast_cache_dump(osm_ucast_mgr_t * p_mgr) { cache_switch_t *p_sw; unsigned i; @@ -380,8 +380,8 @@ static void __ucast_cache_dump(osm_ucast_mgr_t * p_mgr) OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, "\t Switch lid %u %s%s\n", - __cache_sw_get_base_lid_ho(p_sw), - (__cache_sw_is_leaf(p_sw)) ? "[leaf switch] " : "", + cache_sw_get_base_lid_ho(p_sw), + (cache_sw_is_leaf(p_sw)) ? "[leaf switch] " : "", (p_sw->dropped) ? "[whole switch missing]" : ""); for (i = 1; i < p_sw->num_ports; i++) @@ -418,7 +418,7 @@ void osm_ucast_cache_invalidate(osm_ucast_mgr_t * p_mgr) (cache_switch_t *) cl_qmap_end(&p_mgr->cache_sw_tbl)) { p_sw = p_next_sw; p_next_sw = (cache_switch_t *) cl_qmap_next(&p_sw->map_item); - __cache_sw_destroy(p_sw); + cache_sw_destroy(p_sw); } cl_qmap_remove_all(&p_mgr->cache_sw_tbl); Exit: @@ -469,7 +469,7 @@ static void ucast_cache_validate(osm_ucast_mgr_t * p_mgr) p_node = p_sw->p_node; lid_ho = cl_ntoh16(osm_node_get_base_lid(p_node, 0)); - p_cache_sw = __cache_get_sw(p_mgr, lid_ho); + p_cache_sw = cache_get_sw(p_mgr, lid_ho); max_ports = osm_node_get_num_physp(p_node); @@ -490,11 +490,11 @@ static void ucast_cache_validate(osm_ucast_mgr_t * p_mgr) */ if (p_cache_sw && !p_cache_sw->dropped && - !__cache_sw_is_leaf(p_cache_sw) && + !cache_sw_is_leaf(p_cache_sw) && p_physp->p_remote_physp->p_node && osm_node_get_type(p_physp->p_remote_physp-> p_node) != IB_NODE_TYPE_SWITCH) - __cache_sw_set_leaf(p_cache_sw); + cache_sw_set_leaf(p_cache_sw); if (!p_physp->need_update) continue; @@ -657,9 +657,9 @@ static void ucast_cache_validate(osm_ucast_mgr_t * p_mgr) p_cache_sw->ports[port_num].remote_lid_ho = 0; p_cache_sw->ports[port_num].is_leaf = FALSE; if (p_cache_sw->dropped) { - __cache_restore_ucast_info(p_mgr, - p_cache_sw, - p_sw); + cache_restore_ucast_info(p_mgr, + p_cache_sw, + p_sw); p_cache_sw->dropped = FALSE; } @@ -672,7 +672,7 @@ static void ucast_cache_validate(osm_ucast_mgr_t * p_mgr) /* Remove all the cached switches that have all their ports restored */ - __cache_cleanup_switches(p_mgr); + cache_cleanup_switches(p_mgr); /* * Done scanning all the physical switch ports in the subnet. @@ -695,11 +695,11 @@ static void ucast_cache_validate(osm_ucast_mgr_t * p_mgr) (cache_switch_t *) cl_qmap_next(&p_cache_sw->map_item)) { if (p_cache_sw->dropped) { - if (!__cache_sw_is_leaf(p_cache_sw)) { + if (!cache_sw_is_leaf(p_cache_sw)) { OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, "Missing non-leaf switch (lid %u) - " "cache is invalid\n", - __cache_sw_get_base_lid_ho(p_cache_sw)); + cache_sw_get_base_lid_ho(p_cache_sw)); osm_ucast_cache_invalidate(p_mgr); goto Exit; } @@ -707,7 +707,7 @@ static void ucast_cache_validate(osm_ucast_mgr_t * p_mgr) OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, "Missing leaf switch (lid %u) - " "continuing validation\n", - __cache_sw_get_base_lid_ho(p_cache_sw)); + cache_sw_get_base_lid_ho(p_cache_sw)); continue; } @@ -716,35 +716,35 @@ static void ucast_cache_validate(osm_ucast_mgr_t * p_mgr) continue; if (p_cache_sw->ports[port_num].is_leaf) { - CL_ASSERT(__cache_sw_is_leaf(p_cache_sw)); + CL_ASSERT(cache_sw_is_leaf(p_cache_sw)); OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, "Switch lid %u, port %u: missing link to CA/RTR - " "continuing validation\n", - __cache_sw_get_base_lid_ho(p_cache_sw), + cache_sw_get_base_lid_ho(p_cache_sw), port_num); continue; } - p_remote_cache_sw = __cache_get_sw(p_mgr, - p_cache_sw-> - ports[port_num]. - remote_lid_ho); + p_remote_cache_sw = cache_get_sw(p_mgr, + p_cache_sw-> + ports[port_num]. + remote_lid_ho); if (!p_remote_cache_sw || !p_remote_cache_sw->dropped) { OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, "Switch lid %u, port %u: missing link to existing switch - " "cache is invalid\n", - __cache_sw_get_base_lid_ho(p_cache_sw), + cache_sw_get_base_lid_ho(p_cache_sw), port_num); osm_ucast_cache_invalidate(p_mgr); goto Exit; } - if (!__cache_sw_is_leaf(p_remote_cache_sw)) { + if (!cache_sw_is_leaf(p_remote_cache_sw)) { OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, "Switch lid %u, port %u: missing link to non-leaf switch - " "cache is invalid\n", - __cache_sw_get_base_lid_ho(p_cache_sw), + cache_sw_get_base_lid_ho(p_cache_sw), port_num); osm_ucast_cache_invalidate(p_mgr); goto Exit; @@ -759,12 +759,12 @@ static void ucast_cache_validate(osm_ucast_mgr_t * p_mgr) * that use leaf switches to get from switch to switch * and not just to get to the CAs behind the leaf switch. */ - if (__cache_sw_is_leaf(p_cache_sw) && - __cache_sw_is_leaf(p_remote_cache_sw)) { + if (cache_sw_is_leaf(p_cache_sw) && + cache_sw_is_leaf(p_remote_cache_sw)) { OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, "Switch lid %u, port %u: missing leaf-2-leaf link - " "cache is invalid\n", - __cache_sw_get_base_lid_ho(p_cache_sw), + cache_sw_get_base_lid_ho(p_cache_sw), port_num); osm_ucast_cache_invalidate(p_mgr); goto Exit; @@ -773,13 +773,13 @@ static void ucast_cache_validate(osm_ucast_mgr_t * p_mgr) OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, "Switch lid %u, port %u: missing remote leaf switch - " "continuing validation\n", - __cache_sw_get_base_lid_ho(p_cache_sw), + cache_sw_get_base_lid_ho(p_cache_sw), port_num); } } OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, "Unicast cache is valid\n"); - __ucast_cache_dump(p_mgr); + ucast_cache_dump(p_mgr); Exit: OSM_LOG_EXIT(p_mgr->p_log); } /* osm_ucast_cache_validate() */ @@ -799,9 +799,9 @@ void osm_ucast_cache_check_new_link(osm_ucast_mgr_t * p_mgr, if (!p_mgr->cache_valid) goto Exit; - __cache_check_link_change(p_mgr, - osm_node_get_physp_ptr(p_node_1, port_num_1), - osm_node_get_physp_ptr(p_node_2, port_num_2)); + cache_check_link_change(p_mgr, + osm_node_get_physp_ptr(p_node_1, port_num_1), + osm_node_get_physp_ptr(p_node_2, port_num_2)); if (!p_mgr->cache_valid) goto Exit; @@ -845,15 +845,15 @@ void osm_ucast_cache_check_new_link(osm_ucast_mgr_t * p_mgr, goto Exit; } - __cache_remove_port(p_mgr, lid_ho_1, port_num_1, lid_ho_2, - (osm_node_get_type(p_node_2) != - IB_NODE_TYPE_SWITCH)); + cache_remove_port(p_mgr, lid_ho_1, port_num_1, lid_ho_2, + (osm_node_get_type(p_node_2) != + IB_NODE_TYPE_SWITCH)); /* if node_2 is a switch, the link should be cleaned from its cache */ if (osm_node_get_type(p_node_2) == IB_NODE_TYPE_SWITCH) - __cache_remove_port(p_mgr, lid_ho_2, - port_num_2, lid_ho_1, FALSE); + cache_remove_port(p_mgr, lid_ho_2, + port_num_2, lid_ho_1, FALSE); Exit: OSM_LOG_EXIT(p_mgr->p_log); @@ -924,13 +924,13 @@ void osm_ucast_cache_add_link(osm_ucast_mgr_t * p_mgr, lid_ho_2 = cl_ntoh16(osm_node_get_base_lid(p_node_2, 0)); /* lost switch-2-switch link - cache both sides */ - __cache_add_sw_link(p_mgr, p_physp1, lid_ho_2, FALSE); - __cache_add_sw_link(p_mgr, p_physp2, lid_ho_1, FALSE); + cache_add_sw_link(p_mgr, p_physp1, lid_ho_2, FALSE); + cache_add_sw_link(p_mgr, p_physp2, lid_ho_1, FALSE); } else { lid_ho_2 = cl_ntoh16(osm_physp_get_base_lid(p_physp2)); /* lost link to CA/RTR - cache only switch side */ - __cache_add_sw_link(p_mgr, p_physp1, lid_ho_2, TRUE); + cache_add_sw_link(p_mgr, p_physp1, lid_ho_2, TRUE); } Exit: @@ -989,10 +989,10 @@ void osm_ucast_cache_add_node(osm_ucast_mgr_t * p_mgr, osm_node_t * p_node) * and cache can't handle it. */ - p_cache_sw = __cache_get_sw(p_mgr, lid_ho); + p_cache_sw = cache_get_sw(p_mgr, lid_ho); CL_ASSERT(p_cache_sw); - if (!__cache_sw_is_leaf(p_cache_sw)) { + if (!cache_sw_is_leaf(p_cache_sw)) { OSM_LOG(p_mgr->p_log, OSM_LOG_INFO, "Dropped non-leaf switch (lid %u) - " "cache is invalid\n", lid_ho); From sashak at voltaire.com Tue Mar 24 11:06:50 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Mar 2009 20:06:50 +0200 Subject: [ofa-general] Re: [PATCH][TRIVIAL] libibmad/dump.c: Cosmetic formatting changes In-Reply-To: <20090324121642.GA23935@comcast.net> References: <20090324121642.GA23935@comcast.net> Message-ID: <20090324180641.GA20085@sashak.voltaire.com> On 07:16 Tue 24 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Tue Mar 24 11:07:54 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Mar 2009 20:07:54 +0200 Subject: [ofa-general] Re: [PATCH] opensm/iba/ib_types.h: Add PortXmit/RcvDataSL PerfMgt attributes In-Reply-To: <20090324121025.GA23604@comcast.net> References: <20090324121025.GA23604@comcast.net> Message-ID: <20090324180754.GB20085@sashak.voltaire.com> On 07:10 Tue 24 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Tue Mar 24 11:11:11 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Mar 2009 20:11:11 +0200 Subject: [ofa-general] Re: [PATCH] opensm/osm_sminfo_rcv.c: Minor simplification In-Reply-To: <20090324161413.GA17598@comcast.net> References: <20090324161413.GA17598@comcast.net> Message-ID: <20090324181111.GC20085@sashak.voltaire.com> On 11:14 Tue 24 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Tue Mar 24 11:12:52 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Mar 2009 20:12:52 +0200 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm/osm_sm_state_mgr.c: Remove unneeded return statement In-Reply-To: <20090324161501.GB17598@comcast.net> References: <20090324161501.GB17598@comcast.net> Message-ID: <20090324181252.GD20085@sashak.voltaire.com> On 11:15 Tue 24 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Tue Mar 24 11:14:41 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Mar 2009 20:14:41 +0200 Subject: [ofa-general] Re: [PATCH][TRIVIAL] opensm: Remove some __ prefixes In-Reply-To: <20090324184448.GA18569@comcast.net> References: <20090324184448.GA18569@comcast.net> Message-ID: <20090324181441.GE20085@sashak.voltaire.com> On 13:44 Tue 24 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock Applied. Thanks. Sasha From sashak at voltaire.com Tue Mar 24 11:22:24 2009 From: sashak at voltaire.com (Sasha Khapyorsky) Date: Tue, 24 Mar 2009 20:22:24 +0200 Subject: [ofa-general] Re: [PATCH] ibsim: Add SMSL support to PortInfo attribute In-Reply-To: <20090324182510.GA18072@comcast.net> References: <20090324182510.GA18072@comcast.net> Message-ID: <20090324182213.GF20085@sashak.voltaire.com> Hi Hal, On 13:25 Tue 24 Mar , Hal Rosenstock wrote: > > Signed-off-by: Hal Rosenstock > > --- > diff --git a/ibsim/sim.h b/ibsim/sim.h > index 5e9b4f0..4c1439c 100644 > --- a/ibsim/sim.h > +++ b/ibsim/sim.h > @@ -1,5 +1,6 @@ > /* > * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2009 HNR Consulting. All rights reserved. > * > * This file is part of ibsim. > * > @@ -189,6 +190,7 @@ struct Port { > int portnum; > int lid; > int smlid; > + int smsl; What is a purpose of this? Do you have any plans to use this field? If no, I don't see what this patch adds - SMSL is handled already as part of PortInfo buffer. Sasha > int linkwidth; > int linkwidthena; > int linkspeed; > diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c > index d2c313c..a24e5d3 100644 > --- a/ibsim/sim_mad.c > +++ b/ibsim/sim_mad.c > @@ -432,6 +432,7 @@ do_portinfo(Port * port, unsigned op, uint32_t portnum, uint8_t * data) > } > p->lid = newlid; > p->smlid = mad_get_field(data, 0, IB_PORT_SMLID_F); > + p->smsl = mad_get_field(data, 0, IB_PORT_SMSL_F); > // p->linkwidth = mad_get_field(data, 0, IB_PORT_LINK_WIDTH_ENABLED_F); // ignored > p->lmc = mad_get_field(data, 0, IB_PORT_LMC_F); > p->hoqlife = mad_get_field(data, 0, IB_PORT_HOQ_LIFE_F); > diff --git a/ibsim/sim_net.c b/ibsim/sim_net.c > index 13c3b8c..8b95947 100644 > --- a/ibsim/sim_net.c > +++ b/ibsim/sim_net.c > @@ -1,5 +1,6 @@ > /* > * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. > + * Copyright (c) 2009 HNR Consulting. All rights reserved. > * > * This file is part of ibsim. > * > @@ -1077,6 +1078,7 @@ void update_portinfo(Port * p) > p->node->type == SWITCH_NODE ? 0 : p->portnum); > mad_set_field(pi, 0, IB_PORT_LID_F, p->lid); > mad_set_field(pi, 0, IB_PORT_SMLID_F, p->smlid); > + mad_set_field(pi, 0, IB_PORT_SMSL_F, p->smsl); > mad_set_field(pi, 0, IB_PORT_OPER_VLS_F, p->op_vls); > mad_set_field(pi, 0, IB_PORT_LINK_WIDTH_ENABLED_F, p->linkwidthena); > mad_set_field(pi, 0, IB_PORT_LINK_WIDTH_SUPPORTED_F, From hal.rosenstock at gmail.com Tue Mar 24 11:39:19 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 24 Mar 2009 14:39:19 -0400 Subject: ***SPAM*** Re: [ofa-general] Re: [PATCH] ibsim: Add SMSL support to PortInfo attribute In-Reply-To: <20090324182213.GF20085@sashak.voltaire.com> References: <20090324182510.GA18072@comcast.net> <20090324182213.GF20085@sashak.voltaire.com> Message-ID: Hi Sasha, On Tue, Mar 24, 2009 at 2:22 PM, Sasha Khapyorsky wrote: > Hi Hal, > > On 13:25 Tue 24 Mar     , Hal Rosenstock wrote: >> >> Signed-off-by: Hal Rosenstock >> >> --- >> diff --git a/ibsim/sim.h b/ibsim/sim.h >> index 5e9b4f0..4c1439c 100644 >> --- a/ibsim/sim.h >> +++ b/ibsim/sim.h >> @@ -1,5 +1,6 @@ >>  /* >>   * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. >> + * Copyright (c) 2009 HNR Consulting. All rights reserved. >>   * >>   * This file is part of ibsim. >>   * >> @@ -189,6 +190,7 @@ struct Port { >>       int portnum; >>       int lid; >>       int smlid; >> +     int smsl; > > What is a purpose of this? Do you have any plans to use this field? > > If no, I don't see what this patch adds - SMSL is handled already as part > of PortInfo buffer. It's needed when SMSL is not 0 (e.g. Line's recent patch for lash). -- Hal > Sasha > >>       int linkwidth; >>       int linkwidthena; >>       int linkspeed; >> diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c >> index d2c313c..a24e5d3 100644 >> --- a/ibsim/sim_mad.c >> +++ b/ibsim/sim_mad.c >> @@ -432,6 +432,7 @@ do_portinfo(Port * port, unsigned op, uint32_t portnum, uint8_t * data) >>               } >>               p->lid = newlid; >>               p->smlid = mad_get_field(data, 0, IB_PORT_SMLID_F); >> +             p->smsl = mad_get_field(data, 0, IB_PORT_SMSL_F); >>  //              p->linkwidth = mad_get_field(data, 0, IB_PORT_LINK_WIDTH_ENABLED_F); // ignored >>               p->lmc = mad_get_field(data, 0, IB_PORT_LMC_F); >>               p->hoqlife = mad_get_field(data, 0, IB_PORT_HOQ_LIFE_F); >> diff --git a/ibsim/sim_net.c b/ibsim/sim_net.c >> index 13c3b8c..8b95947 100644 >> --- a/ibsim/sim_net.c >> +++ b/ibsim/sim_net.c >> @@ -1,5 +1,6 @@ >>  /* >>   * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved. >> + * Copyright (c) 2009 HNR Consulting. All rights reserved. >>   * >>   * This file is part of ibsim. >>   * >> @@ -1077,6 +1078,7 @@ void update_portinfo(Port * p) >>                     p->node->type == SWITCH_NODE ? 0 : p->portnum); >>       mad_set_field(pi, 0, IB_PORT_LID_F, p->lid); >>       mad_set_field(pi, 0, IB_PORT_SMLID_F, p->smlid); >> +     mad_set_field(pi, 0, IB_PORT_SMSL_F, p->smsl); >>       mad_set_field(pi, 0, IB_PORT_OPER_VLS_F, p->op_vls); >>       mad_set_field(pi, 0, IB_PORT_LINK_WIDTH_ENABLED_F, p->linkwidthena); >>       mad_set_field(pi, 0, IB_PORT_LINK_WIDTH_SUPPORTED_F, > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From hnrose at comcast.net Tue Mar 24 12:41:21 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 24 Mar 2009 14:41:21 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] opensm/include/ib_types.h: Add ib_switch_info_get_state_opt_sl2vlmapping routine Message-ID: <20090324194121.GA1010@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/include/iba/ib_types.h b/opensm/include/iba/ib_types.h index e148597..71443bd 100644 --- a/opensm/include/iba/ib_types.h +++ b/opensm/include/iba/ib_types.h @@ -5968,6 +5968,34 @@ ib_switch_info_clear_state_change(IN ib_switch_info_t * const p_si) * SEE ALSO *********/ +/****f* IBA Base: Types/ib_switch_info_get_opt_sl2vlmapping +* NAME +* ib_switch_info_get_state_opt_sl2vlmapping +* +* DESCRIPTION +* Returns the value of the optimized SLtoVLMapping programming flag. +* +* SYNOPSIS +*/ +static inline boolean_t OSM_API +ib_switch_info_get_opt_sl2vlmapping(IN const ib_switch_info_t * const p_si) +{ + return ((p_si->life_state & 0x01) == 0x01); +} + +/* +* PARAMETERS +* p_si +* [in] Pointer to a SwitchInfo attribute. +* +* RETURN VALUES +* Returns the value of the optimized SLtoVLMapping programming flag. +* +* NOTES +* +* SEE ALSO +*********/ + /****f* IBA Base: Types/ib_switch_info_is_enhanced_port0 * NAME * ib_switch_info_is_enhanced_port0 From hnrose at comcast.net Tue Mar 24 15:16:55 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Tue, 24 Mar 2009 17:16:55 -0500 Subject: [ofa-general] ***SPAM*** [PATCH][TRIVIAL] opensm/osm_qos.c: Cosmetic formatting changes Message-ID: <20090324221655.GA3495@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_qos.c b/opensm/opensm/osm_qos.c index 25164ae..a8ffc08 100644 --- a/opensm/opensm/osm_qos.c +++ b/opensm/opensm/osm_qos.c @@ -58,8 +58,8 @@ struct qos_config { ib_slvl_table_t sl2vl; }; -static void qos_build_config(struct qos_config *cfg, - osm_qos_options_t * opt, osm_qos_options_t * dflt); +static void qos_build_config(struct qos_config * cfg, osm_qos_options_t * opt, + osm_qos_options_t * dflt); /* * QoS primitives @@ -102,10 +102,9 @@ static ib_api_status_t vlarb_update_table_block(osm_sm_t * sm, cl_hton32(attr_mod), CL_DISP_MSGID_NONE, &context); } -static ib_api_status_t vlarb_update(osm_sm_t * sm, - osm_physp_t * p, uint8_t port_num, - unsigned force_update, - const struct qos_config *qcfg) +static ib_api_status_t vlarb_update(osm_sm_t * sm, osm_physp_t * p, + uint8_t port_num, unsigned force_update, + const struct qos_config * qcfg) { ib_api_status_t status = IB_SUCCESS; ib_port_info_t *p_pi = &p->port_info; @@ -149,9 +148,8 @@ static ib_api_status_t vlarb_update(osm_sm_t * sm, return status; } -static ib_api_status_t sl2vl_update_table(osm_sm_t * sm, - osm_physp_t * p, uint8_t in_port, - uint8_t out_port, +static ib_api_status_t sl2vl_update_table(osm_sm_t * sm, osm_physp_t * p, + uint8_t in_port, uint8_t out_port, unsigned force_update, const ib_slvl_table_t * sl2vl_table) { @@ -184,7 +182,7 @@ static ib_api_status_t sl2vl_update_table(osm_sm_t * sm, context.slvl_context.set_method = TRUE; attr_mod = in_port << 8 | out_port; return osm_req_set(sm, osm_physp_get_dr_path_ptr(p), - (uint8_t *) & tbl, sizeof(tbl), + (uint8_t *) &tbl, sizeof(tbl), IB_MAD_ATTR_SLVL_TABLE, cl_hton32(attr_mod), CL_DISP_MSGID_NONE, &context); } @@ -215,9 +213,8 @@ static ib_api_status_t sl2vl_update(osm_sm_t * sm, osm_port_t * p_port, } for (i = 0; i < num_ports; i++) { - status = - sl2vl_update_table(sm, p, i, port_num, - force_update, &qcfg->sl2vl); + status = sl2vl_update_table(sm, p, i, port_num, force_update, + &qcfg->sl2vl); if (status != IB_SUCCESS) return status; } @@ -227,8 +224,7 @@ static ib_api_status_t sl2vl_update(osm_sm_t * sm, osm_port_t * p_port, static ib_api_status_t qos_physp_setup(osm_log_t * p_log, osm_sm_t * sm, osm_port_t * p_port, osm_physp_t * p, - uint8_t port_num, - unsigned force_update, + uint8_t port_num, unsigned force_update, const struct qos_config *qcfg) { ib_api_status_t status; @@ -330,8 +326,8 @@ osm_signal_t osm_qos_setup(osm_opensm_t * p_osm) continue; force_update = p_physp->need_update || p_osm->subn.need_update; - status = qos_physp_setup(&p_osm->log, &p_osm->sm, - p_port, p_physp, 0, force_update, cfg); + status = qos_physp_setup(&p_osm->log, &p_osm->sm, p_port, + p_physp, 0, force_update, cfg); } cl_plock_release(&p_osm->lock); @@ -373,8 +369,8 @@ static int parse_sl2vl_entry(char *str, uint8_t * raw) return (int)(p - str); } -static void qos_build_config(struct qos_config *cfg, - osm_qos_options_t * opt, osm_qos_options_t * dflt) +static void qos_build_config(struct qos_config * cfg, osm_qos_options_t * opt, + osm_qos_options_t * dflt) { int i; char *p; From rdreier at cisco.com Tue Mar 24 20:33:45 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 24 Mar 2009 20:33:45 -0700 Subject: [ofa-general] RDMA/InfiniBand merge plans for 2.6.30 Message-ID: Since 2.6.29 is out, I should update everyone on 2.6.30 merge plans. All the pending things that I'm aware of are listed below. As usual, if something isn't already in my tree and it isn't listed below, I probably missed it or dropped it unintentionally. Please remind me again. Boilerplate: As usual, when submitting a patch: - Give a good changelog that explains what issue your patch addresses, how you address the issue, how serious the issue is, and any other information that would be useful to someone evaluating your patch or reading it years from now. - Please make sure that you include a "Signed-off-by:" line, and put any extra junk that should not go into the final kernel log *after* the "---" line so that git tools strip it off automatically. Make the subject line be appropriate for inclusion in the kernel log as well once the leading "[PATCH ...]" stuff is stripped off. I waste a lot of time fixing patches by hand that could otherwise be spent doing something productive like watching youtube. - Run your patch through checkpatch.pl so I don't have to nag you to fix trivial issues (or spend time fixing them myself). - Read your patch over so I don't see a memory leak or deadlock as soon as I look at it. - Build your patch with sparse checking ("C=2 CF=-D__CHECK_ENDIAN__") and make sure it doesn't introduce new warnings. (A big bonus in goodwill for sending patches that fix old warnings) - Test your patch on a kernel with things like slab debugging and lockdep turned on. And while you're waiting for me to get to your patch, I sure wouldn't mind if you read and commented on someone else's patch. None of this means you shouldn't remind me about pending patches, since I often lose track of things and drop them accidentally. Core: - Fix for sysfs lifetime problem that could allow userspace to call into a device driver after it's unloaded. - Several fixes for the MAD handling layer. ULPs: - I have one IPoIB fix merge, and I have the nagging feeling that there are still IPoIB crash bugs that I lost track of. HW specific: - I have a few cxgb3 fixes merged already, and I'll merge the EEH patch after the net-next tree lands, just to avoid merge hassles. - Many nes fixes. - Support for port type (IB or Ethernet) autonegotiation for mlx4 devices. Here are a few topics that I believe will not be ready in time for the 2.6.30 window and will need to wait for 2.6.31 at least: - Jack's XRC patch set. I just need time to work with Jack and clean up the code. Here all the patches I already have in my for-next branch: Chien Tung (3): RDMA/nes: Update copyright to new legal entity and year RDMA/nes: Report correct vendor_id and vendor_part_id RDMA/nes: Fix tmp_addr compilation warning Don Wood (3): RDMA/nes: Account for freed PBL after HW operation RDMA/nes: Inform hardware that asynchronous event has been handled RDMA/nes: Improve use of PBLs Faisal Latif (2): RDMA/nes: Remove LLTX RDMA/nes: Handle MPA Reject message properly Harvey Harrison (1): IB: Remove __constant_{endian} uses Jack Morgenstein (3): IPoIB: In unicast_arp_send(), only free newly-created paths mlx4_core: Add device IDs for MT25458 10GigE devices IB/sa_query: Fix AH leak due to update_sm_ah() race Moni Shoua (2): IB/mlx4: Fix dispatch of IB_EVENT_LID_CHANGE event IB/mthca: Fix dispatch of IB_EVENT_LID_CHANGE event Or Gerlitz (1): IB/iser: Remove hard setting of path MTU Ralph Campbell (3): IB/mad: Fix null pointer dereference in local_completions() IB/mad: initialize mad_agent_priv before putting on lists IB/mad: Fix ib_post_send_mad() returning 0 with no generate send comp Ramachandra K (1): IB/mad: Fix RMPP header RRespTime manipulation Roland Dreier (5): IB/ipath: Fix memory leak in init_shadow_tids() error path IB/ipath: Really run work in ipath_release_user_pages_on_close() IB: Remove sysfs files before unregistering device IB: Remove useless ibdev_is_alive() tests from sysfs code Merge branches 'cxgb3', 'endian', 'ipath', 'ipoib', 'iser', 'mad', 'misc', 'mlx4', 'mthca', 'nes' and 'sysfs' into for-next Steve Wise (3): RDMA/cxgb3: sgl/pbl offset calculation needs 64 bits RDMA/cxgb3: Connection termination fixes RDMA/cxgb3: Remove modulo math from build_rdma_recv() Yevgeny Petrilin (3): mlx4_core: Don't perform SET_PORT command for Ethernet ports mlx4_core: Add link type autosensing IB/mlx4: Unregister IB device prior to CLOSE PORT command From rdreier at cisco.com Tue Mar 24 20:45:35 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 24 Mar 2009 20:45:35 -0700 Subject: [ofa-general] Re: [PATCH 2.6.30] RDMA/cxgb3: Enforce required firmware during device discovery. In-Reply-To: <20090309102114.5779.32122.stgit@build.ogc.int> (Steve Wise's message of "Mon, 09 Mar 2009 05:21:15 -0500") References: <20090309102114.5779.32122.stgit@build.ogc.int> Message-ID: thanks, applied From rdreier at cisco.com Tue Mar 24 21:05:00 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 24 Mar 2009 21:05:00 -0700 Subject: [ofa-general] [GIT PULL] please pull infiniband.git Message-ID: Linus, please pull from master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This tree is also available from kernel.org mirrors at: git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git for-linus This has the first batch of 2.6.30 changes -- I have one more patch that I'll send after the net-next tree is merged (just to avoid merge hassles due to dependencies). Chien Tung (3): RDMA/nes: Update copyright to new legal entity and year RDMA/nes: Report correct vendor_id and vendor_part_id RDMA/nes: Fix tmp_addr compilation warning Don Wood (3): RDMA/nes: Account for freed PBL after HW operation RDMA/nes: Inform hardware that asynchronous event has been handled RDMA/nes: Improve use of PBLs Faisal Latif (2): RDMA/nes: Remove LLTX RDMA/nes: Handle MPA Reject message properly Harvey Harrison (1): IB: Remove __constant_{endian} uses Jack Morgenstein (3): IPoIB: In unicast_arp_send(), only free newly-created paths mlx4_core: Add device IDs for MT25458 10GigE devices IB/sa_query: Fix AH leak due to update_sm_ah() race Moni Shoua (2): IB/mlx4: Fix dispatch of IB_EVENT_LID_CHANGE event IB/mthca: Fix dispatch of IB_EVENT_LID_CHANGE event Or Gerlitz (1): IB/iser: Remove hard setting of path MTU Ralph Campbell (3): IB/mad: Fix null pointer dereference in local_completions() IB/mad: initialize mad_agent_priv before putting on lists IB/mad: Fix ib_post_send_mad() returning 0 with no generate send comp Ramachandra K (1): IB/mad: Fix RMPP header RRespTime manipulation Roland Dreier (5): IB/ipath: Fix memory leak in init_shadow_tids() error path IB/ipath: Really run work in ipath_release_user_pages_on_close() IB: Remove sysfs files before unregistering device IB: Remove useless ibdev_is_alive() tests from sysfs code Merge branches 'cxgb3', 'endian', 'ipath', 'ipoib', 'iser', 'mad', 'misc', 'mlx4', 'mthca', 'nes' and 'sysfs' into for-next Steve Wise (4): RDMA/cxgb3: sgl/pbl offset calculation needs 64 bits RDMA/cxgb3: Connection termination fixes RDMA/cxgb3: Remove modulo math from build_rdma_recv() RDMA/cxgb3: Enforce required firmware Yevgeny Petrilin (3): mlx4_core: Don't perform SET_PORT command for Ethernet ports mlx4_core: Add link type autosensing IB/mlx4: Unregister IB device prior to CLOSE PORT command drivers/infiniband/core/cm.c | 15 +- drivers/infiniband/core/cm_msgs.h | 22 +- drivers/infiniband/core/device.c | 4 +- drivers/infiniband/core/mad.c | 40 +- drivers/infiniband/core/mad_rmpp.c | 2 +- drivers/infiniband/core/sa_query.c | 2 + drivers/infiniband/core/sysfs.c | 19 +- drivers/infiniband/hw/cxgb3/cxio_hal.c | 30 ++- drivers/infiniband/hw/cxgb3/cxio_hal.h | 3 + drivers/infiniband/hw/cxgb3/cxio_wr.h | 6 + drivers/infiniband/hw/cxgb3/iwch_cm.c | 3 + drivers/infiniband/hw/cxgb3/iwch_ev.c | 5 - drivers/infiniband/hw/cxgb3/iwch_qp.c | 17 +- drivers/infiniband/hw/ehca/ehca_sqp.c | 8 +- drivers/infiniband/hw/ipath/ipath_eeprom.c | 4 +- drivers/infiniband/hw/ipath/ipath_init_chip.c | 2 +- drivers/infiniband/hw/ipath/ipath_mad.c | 95 ++-- drivers/infiniband/hw/ipath/ipath_rc.c | 2 +- drivers/infiniband/hw/ipath/ipath_sdma.c | 4 +- drivers/infiniband/hw/ipath/ipath_uc.c | 2 +- drivers/infiniband/hw/ipath/ipath_ud.c | 4 +- drivers/infiniband/hw/ipath/ipath_user_pages.c | 8 +- drivers/infiniband/hw/ipath/ipath_user_sdma.c | 6 +- drivers/infiniband/hw/ipath/ipath_verbs.c | 2 +- drivers/infiniband/hw/ipath/ipath_verbs.h | 10 +- drivers/infiniband/hw/mlx4/mad.c | 27 +- drivers/infiniband/hw/mlx4/main.c | 5 +- drivers/infiniband/hw/mlx4/qp.c | 22 +- drivers/infiniband/hw/mthca/mthca_mad.c | 25 +- drivers/infiniband/hw/nes/nes.c | 2 +- drivers/infiniband/hw/nes/nes.h | 2 +- drivers/infiniband/hw/nes/nes_cm.c | 586 ++++++++++++++++-------- drivers/infiniband/hw/nes/nes_cm.h | 12 +- drivers/infiniband/hw/nes/nes_context.h | 2 +- drivers/infiniband/hw/nes/nes_hw.c | 17 +- drivers/infiniband/hw/nes/nes_hw.h | 5 +- drivers/infiniband/hw/nes/nes_nic.c | 142 +++---- drivers/infiniband/hw/nes/nes_user.h | 2 +- drivers/infiniband/hw/nes/nes_utils.c | 2 +- drivers/infiniband/hw/nes/nes_verbs.c | 249 +++++++---- drivers/infiniband/hw/nes/nes_verbs.h | 2 +- drivers/infiniband/ulp/ipoib/ipoib_main.c | 9 +- drivers/infiniband/ulp/iser/iser_verbs.c | 7 - drivers/net/mlx4/Makefile | 2 +- drivers/net/mlx4/catas.c | 16 +- drivers/net/mlx4/eq.c | 16 +- drivers/net/mlx4/main.c | 106 ++++-- drivers/net/mlx4/mlx4.h | 27 +- drivers/net/mlx4/port.c | 13 +- drivers/net/mlx4/sense.c | 156 +++++++ include/linux/mlx4/cmd.h | 1 + include/linux/mlx4/device.h | 6 +- include/rdma/ib_cm.h | 12 +- include/rdma/ib_mad.h | 4 +- include/rdma/ib_smi.h | 34 +- 55 files changed, 1190 insertions(+), 636 deletions(-) create mode 100644 drivers/net/mlx4/sense.c From diego.guella at sircomtech.com Wed Mar 25 01:59:55 2009 From: diego.guella at sircomtech.com (Diego Guella) Date: Wed, 25 Mar 2009 09:59:55 +0100 Subject: [ofa-general] RE: [ofw] What is the current support level for QoS in WinOF? References: <018301c9aba1$0a0c4370$05c8a8c0@DIEGO> <18C53B9CBFF14CDB930F6E52B77BD9D0@amr.corp.intel.com> <008801c9ac52$f3abad20$05c8a8c0@DIEGO> Message-ID: <002901c9ad28$1139fd80$05c8a8c0@DIEGO> >From: "Hal Rosenstock" >There are 2 docs: >qos-config.txt (describes "primitive QoS support") and >QoS_management_in_OpenSM.txt (describes QoS Annex support). Is qos-config.txt describing the current QoS support level in WinOF? Where can I find that file? Googling returns this: https://svn.openfabrics.org/svn/openib/gen2/branches/1.1/src/userspace/management/osm/doc/qos-config.txt From vlad at lists.openfabrics.org Wed Mar 25 03:29:04 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Wed, 25 Mar 2009 03:29:04 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090325-0200 daily build status Message-ID: <20090325102904.644F6E61185@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From hal.rosenstock at gmail.com Wed Mar 25 03:34:23 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Wed, 25 Mar 2009 06:34:23 -0400 Subject: ***SPAM*** Re: [ofa-general] RE: [ofw] What is the current support level for QoS in WinOF? In-Reply-To: <002901c9ad28$1139fd80$05c8a8c0@DIEGO> References: <018301c9aba1$0a0c4370$05c8a8c0@DIEGO> <18C53B9CBFF14CDB930F6E52B77BD9D0@amr.corp.intel.com> <008801c9ac52$f3abad20$05c8a8c0@DIEGO> <002901c9ad28$1139fd80$05c8a8c0@DIEGO> Message-ID: On Wed, Mar 25, 2009 at 4:59 AM, Diego Guella wrote: >> From: "Hal Rosenstock" >> There are 2 docs: >> qos-config.txt (describes "primitive QoS support") and >> QoS_management_in_OpenSM.txt (describes QoS Annex support). > > Is qos-config.txt describing the current QoS support level in WinOF? > Where can I find that file? > Googling returns this: > https://svn.openfabrics.org/svn/openib/gen2/branches/1.1/src/userspace/management/osm/doc/qos-config.txt Yes, that's the doc. -- Hal From arnd at arndb.de Wed Mar 25 05:58:05 2009 From: arnd at arndb.de (Arnd Bergmann) Date: Wed, 25 Mar 2009 13:58:05 +0100 Subject: [ofa-general] Re: RDMA/InfiniBand merge plans for 2.6.30 In-Reply-To: References: Message-ID: <200903251358.06128.arnd@arndb.de> On Wednesday 25 March 2009, Roland Dreier wrote: > Since 2.6.29 is out, I should update everyone on 2.6.30 merge plans. > All the pending things that I'm aware of are listed below. As usual, if > something isn't already in my tree and it isn't listed below, I probably > missed it or dropped it unintentionally. Please remind me again. There are some patches in the ofed release by Eli and me to support weak ordering on DMA ("ib_core: Use weak ordering for data registered memory"). When they got merged into OFED, I assumed they would find their way into mainline along with most of the other patches in there. Did they get forgotten about, or did I misunderstand the process? Arnd <>< From rdreier at cisco.com Wed Mar 25 07:15:58 2009 From: rdreier at cisco.com (Roland Dreier) Date: Wed, 25 Mar 2009 07:15:58 -0700 Subject: [ofa-general] Re: RDMA/InfiniBand merge plans for 2.6.30 In-Reply-To: <200903251358.06128.arnd@arndb.de> (Arnd Bergmann's message of "Wed, 25 Mar 2009 13:58:05 +0100") References: <200903251358.06128.arnd@arndb.de> Message-ID: > There are some patches in the ofed release by Eli and me to support > weak ordering on DMA ("ib_core: Use weak ordering for data registered > memory"). When they got merged into OFED, I assumed they would find > their way into mainline along with most of the other patches in there. > > Did they get forgotten about, or did I misunderstand the process? Oh yeah, forgot about that weak ordering. I still want to have a think about that (I don't like having a module parameter for it but maybe there's not better way) However in general getting into OFED is pretty unrelated to getting upstream... - R. From hnrose at comcast.net Wed Mar 25 08:21:53 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 25 Mar 2009 10:21:53 -0500 Subject: [ofa-general] [PATCH][TRIVIAL] opensm/osm_slvl_map_rcv.c: Cosmetic formatting changes Message-ID: <20090325152153.GB8376@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_slvl_map_rcv.c b/opensm/opensm/osm_slvl_map_rcv.c index f5a7370..9c37442 100644 --- a/opensm/opensm/osm_slvl_map_rcv.c +++ b/opensm/opensm/osm_slvl_map_rcv.c @@ -97,8 +97,7 @@ void osm_slvl_rcv_process(IN void *context, IN void *p_data) OSM_LOG(sm->p_log, OSM_LOG_ERROR, "ERR 2C06: " "No port object for port with GUID 0x%" PRIx64 "\n\t\t\t\tfor parent node GUID 0x%" PRIx64 - ", TID 0x%" PRIx64 "\n", - cl_ntoh64(port_guid), + ", TID 0x%" PRIx64 "\n", cl_ntoh64(port_guid), cl_ntoh64(node_guid), cl_ntoh64(p_smp->trans_id)); goto Exit; } @@ -139,8 +138,7 @@ void osm_slvl_rcv_process(IN void *context, IN void *p_data) goto Exit; } - osm_dump_slvl_map_table(sm->p_log, - port_guid, in_port_num, + osm_dump_slvl_map_table(sm->p_log, port_guid, in_port_num, out_port_num, p_slvl_tbl, OSM_LOG_DEBUG); osm_physp_set_slvl_tbl(p_physp, p_slvl_tbl, in_port_num); From hnrose at comcast.net Wed Mar 25 08:21:03 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 25 Mar 2009 10:21:03 -0500 Subject: [ofa-general] ***SPAM*** [PATCHv2][TRIVIAL] opensm/osm_qos.c: Cosmetic formatting changes Message-ID: <20090325152103.GA8376@comcast.net> Signed-off-by: Hal Rosenstock --- Changes since v1: More formatting changes diff --git a/opensm/opensm/osm_qos.c b/opensm/opensm/osm_qos.c index 25164ae..fb311ca 100644 --- a/opensm/opensm/osm_qos.c +++ b/opensm/opensm/osm_qos.c @@ -58,8 +58,8 @@ struct qos_config { ib_slvl_table_t sl2vl; }; -static void qos_build_config(struct qos_config *cfg, - osm_qos_options_t * opt, osm_qos_options_t * dflt); +static void qos_build_config(struct qos_config * cfg, osm_qos_options_t * opt, + osm_qos_options_t * dflt); /* * QoS primitives @@ -97,15 +97,14 @@ static ib_api_status_t vlarb_update_table_block(osm_sm_t * sm, attr_mod = ((block_num + 1) << 16) | port_num; return osm_req_set(sm, osm_physp_get_dr_path_ptr(p), - (uint8_t *) & block, sizeof(block), - IB_MAD_ATTR_VL_ARBITRATION, - cl_hton32(attr_mod), CL_DISP_MSGID_NONE, &context); + (uint8_t *) &block, sizeof(block), + IB_MAD_ATTR_VL_ARBITRATION, cl_hton32(attr_mod), + CL_DISP_MSGID_NONE, &context); } -static ib_api_status_t vlarb_update(osm_sm_t * sm, - osm_physp_t * p, uint8_t port_num, - unsigned force_update, - const struct qos_config *qcfg) +static ib_api_status_t vlarb_update(osm_sm_t * sm, osm_physp_t * p, + uint8_t port_num, unsigned force_update, + const struct qos_config * qcfg) { ib_api_status_t status = IB_SUCCESS; ib_port_info_t *p_pi = &p->port_info; @@ -149,9 +148,8 @@ static ib_api_status_t vlarb_update(osm_sm_t * sm, return status; } -static ib_api_status_t sl2vl_update_table(osm_sm_t * sm, - osm_physp_t * p, uint8_t in_port, - uint8_t out_port, +static ib_api_status_t sl2vl_update_table(osm_sm_t * sm, osm_physp_t * p, + uint8_t in_port, uint8_t out_port, unsigned force_update, const ib_slvl_table_t * sl2vl_table) { @@ -184,9 +182,9 @@ static ib_api_status_t sl2vl_update_table(osm_sm_t * sm, context.slvl_context.set_method = TRUE; attr_mod = in_port << 8 | out_port; return osm_req_set(sm, osm_physp_get_dr_path_ptr(p), - (uint8_t *) & tbl, sizeof(tbl), - IB_MAD_ATTR_SLVL_TABLE, - cl_hton32(attr_mod), CL_DISP_MSGID_NONE, &context); + (uint8_t *) &tbl, sizeof(tbl), + IB_MAD_ATTR_SLVL_TABLE, cl_hton32(attr_mod), + CL_DISP_MSGID_NONE, &context); } static ib_api_status_t sl2vl_update(osm_sm_t * sm, osm_port_t * p_port, @@ -215,9 +213,8 @@ static ib_api_status_t sl2vl_update(osm_sm_t * sm, osm_port_t * p_port, } for (i = 0; i < num_ports; i++) { - status = - sl2vl_update_table(sm, p, i, port_num, - force_update, &qcfg->sl2vl); + status = sl2vl_update_table(sm, p, i, port_num, force_update, + &qcfg->sl2vl); if (status != IB_SUCCESS) return status; } @@ -227,8 +224,7 @@ static ib_api_status_t sl2vl_update(osm_sm_t * sm, osm_port_t * p_port, static ib_api_status_t qos_physp_setup(osm_log_t * p_log, osm_sm_t * sm, osm_port_t * p_port, osm_physp_t * p, - uint8_t port_num, - unsigned force_update, + uint8_t port_num, unsigned force_update, const struct qos_config *qcfg) { ib_api_status_t status; @@ -330,8 +326,8 @@ osm_signal_t osm_qos_setup(osm_opensm_t * p_osm) continue; force_update = p_physp->need_update || p_osm->subn.need_update; - status = qos_physp_setup(&p_osm->log, &p_osm->sm, - p_port, p_physp, 0, force_update, cfg); + status = qos_physp_setup(&p_osm->log, &p_osm->sm, p_port, + p_physp, 0, force_update, cfg); } cl_plock_release(&p_osm->lock); @@ -373,8 +369,8 @@ static int parse_sl2vl_entry(char *str, uint8_t * raw) return (int)(p - str); } -static void qos_build_config(struct qos_config *cfg, - osm_qos_options_t * opt, osm_qos_options_t * dflt) +static void qos_build_config(struct qos_config * cfg, osm_qos_options_t * opt, + osm_qos_options_t * dflt) { int i; char *p; From hnrose at comcast.net Wed Mar 25 08:22:46 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Wed, 25 Mar 2009 10:22:46 -0500 Subject: [ofa-general] ***SPAM*** [PATCH][TRIVIAL] opensm/osm_link_mgr.c: Remove extraneous parentheses Message-ID: <20090325152246.GA8386@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_link_mgr.c b/opensm/opensm/osm_link_mgr.c index b785126..d189471 100644 --- a/opensm/opensm/osm_link_mgr.c +++ b/opensm/opensm/osm_link_mgr.c @@ -155,9 +155,8 @@ static boolean_t link_mgr_set_physp_pi(osm_sm_t * sm, IN osm_physp_t * p_physp, (p_physp))); goto Exit; } - } else { + } else esp0 = TRUE; - } } memset(payload, 0, IB_SMP_DATA_SIZE); From eli at mellanox.co.il Wed Mar 25 07:51:14 2009 From: eli at mellanox.co.il (Eli Cohen) Date: Wed, 25 Mar 2009 16:51:14 +0200 Subject: [ofa-general] [PATCH] IB/ipoib: set neigh->dgid upon ipoib_neigh creation Message-ID: <20090325145114.GA31206@mtls03> If we fail to do that, there can be superfluous calls to ipoib_neigh_free()/ipoib_neigh_alloc() due to the following if statement in ipoib_start_xmit(): if (unlikely((memcmp(&neigh->dgid.raw, skb->dst->neighbour->ha + 4, sizeof(union ib_gid))) || In the case of a unicast GID, it can happen until the path is resolved and is not so severe. In the case of a multicast address, neigh->dgid is never even updated, so the corresponding ipoib_neigh will be destroyed and created on every multicast packet sent. Signed-off-by: Eli Cohen --- drivers/infiniband/ulp/ipoib/ipoib_main.c | 3 +-- 1 files changed, 1 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index 0bd2a4f..0005107 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -460,8 +460,6 @@ static void path_rec_completion(int status, } kref_get(&path->ah->ref); neigh->ah = path->ah; - memcpy(&neigh->dgid.raw, &path->pathrec.dgid.raw, - sizeof(union ib_gid)); if (ipoib_cm_enabled(dev, neigh->neighbour)) { if (!ipoib_cm_get(neigh)) @@ -878,6 +876,7 @@ struct ipoib_neigh *ipoib_neigh_alloc(struct neighbour *neighbour, neigh->neighbour = neighbour; neigh->dev = dev; *to_ipoib_neigh(neighbour) = neigh; + memcpy(neigh->dgid.raw, neighbour->ha + 4, 16); skb_queue_head_init(&neigh->queue); ipoib_cm_set(neigh, NULL); -- 1.6.2.1 From tziporet at dev.mellanox.co.il Wed Mar 25 08:38:53 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Wed, 25 Mar 2009 08:38:53 -0700 Subject: [ofa-general] RDMA/InfiniBand merge plans for 2.6.30 In-Reply-To: References: Message-ID: <49CA500D.20603@mellanox.co.il> Roland Dreier wrote: > Here are a few topics that I believe will not be ready in time for the > 2.6.30 window and will need to wait for 2.6.31 at least: > > - Jack's XRC patch set. I just need time to work with Jack and clean > up the code. > Any chance we can make it for 2.6.30? Tziporet From yosefe at Voltaire.COM Wed Mar 25 12:20:19 2009 From: yosefe at Voltaire.COM (Yossi Etigin) Date: Wed, 25 Mar 2009 21:20:19 +0200 Subject: [ofa-general] Re: [PATCH] IB/ipoib: set neigh->dgid upon ipoib_neigh creation In-Reply-To: <20090325145114.GA31206@mtls03> References: <20090325145114.GA31206@mtls03> Message-ID: <49CA83F3.9040204@Voltaire.COM> Hi Eli, Wouldn't this patch break the fast bond+SM failover recovery? Up till now, in such case neigh->dgid was the old gid until path record query was successful, and that caused to retry the path query until it's successful. With that patch, a new path query will not be issued until the next arp refresh because neigh->dgid will be set to a valid value right after the first packet. How about keeping the memcpy in path_rec-completion, and also adding it to ib_mcast_send (where ipoib neigh is created)? --Yossi Eli Cohen wrote: > If we fail to do that, there can be superfluous calls to > ipoib_neigh_free()/ipoib_neigh_alloc() due to the following if statement in > ipoib_start_xmit(): > > if (unlikely((memcmp(&neigh->dgid.raw, > skb->dst->neighbour->ha + 4, > sizeof(union ib_gid))) || > > In the case of a unicast GID, it can happen until the path is resolved and is > not so severe. In the case of a multicast address, neigh->dgid is never even > updated, so the corresponding ipoib_neigh will be destroyed and created on > every multicast packet sent. > > Signed-off-by: Eli Cohen > --- > drivers/infiniband/ulp/ipoib/ipoib_main.c | 3 +-- > 1 files changed, 1 insertions(+), 2 deletions(-) > > diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c > index 0bd2a4f..0005107 100644 > --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c > +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c > @@ -460,8 +460,6 @@ static void path_rec_completion(int status, > } > kref_get(&path->ah->ref); > neigh->ah = path->ah; > - memcpy(&neigh->dgid.raw, &path->pathrec.dgid.raw, > - sizeof(union ib_gid)); > > if (ipoib_cm_enabled(dev, neigh->neighbour)) { > if (!ipoib_cm_get(neigh)) > @@ -878,6 +876,7 @@ struct ipoib_neigh *ipoib_neigh_alloc(struct neighbour *neighbour, > neigh->neighbour = neighbour; > neigh->dev = dev; > *to_ipoib_neigh(neighbour) = neigh; > + memcpy(neigh->dgid.raw, neighbour->ha + 4, 16); > skb_queue_head_init(&neigh->queue); > ipoib_cm_set(neigh, NULL); > From yosefe at Voltaire.COM Wed Mar 25 12:35:47 2009 From: yosefe at Voltaire.COM (Yossi Etigin) Date: Wed, 25 Mar 2009 21:35:47 +0200 Subject: [ofa-general] [PATCH] ipoib: take rtnl_lock when calling ipoib_open directly Message-ID: <49CA8793.6010409@Voltaire.COM> ipoib_open() needs to be called with rtnl_lock because it's normally called by netlink with rtnl_lock taken. Moreover, it calls dev_change_flags() which needs rtnl_lock. Signed-off-by: Yossi Etigin -- Index: b/drivers/infiniband/ulp/ipoib/ipoib_ib.c =================================================================== --- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c 2009-03-10 18:21:47.000000000 +0200 +++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c 2009-03-10 19:21:22.000000000 +0200 @@ -1039,8 +1039,11 @@ void ipoib_pkey_poll(struct work_struct ipoib_pkey_dev_check_presence(dev); - if (test_bit(IPOIB_PKEY_ASSIGNED, &priv->flags)) + if (test_bit(IPOIB_PKEY_ASSIGNED, &priv->flags)) { + rtnl_lock(); ipoib_open(dev); + rtnl_unlock(); + } else { mutex_lock(&pkey_mutex); if (!test_bit(IPOIB_PKEY_STOP, &priv->flags)) From yosefe at Voltaire.COM Wed Mar 25 12:40:23 2009 From: yosefe at Voltaire.COM (Yossi Etigin) Date: Wed, 25 Mar 2009 21:40:23 +0200 Subject: [ofa-general] [PATCH] rdma_cm: create cm id even when port is down Message-ID: <49CA88A7.7060006@Voltaire.COM> When doing rdma_resolve_addr() and relevant port is down, the function fails and rdma_cm id is not bound to the device. Therefore, application does not have device handle and cannot wait for the port to become active. The function fails because ipoib is not joined to the multicast group and therefore sa does not have a multicast record to take a qkey from. The patch here is to make lazy qkey resolution - cma_set_qkey will set id_priv->qkey if it was not set, and will be called just before the qkey is really required. Signed-off-by: Yossi Etigin --- drivers/infiniband/core/cma.c | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-) Index: b/drivers/infiniband/core/cma.c =================================================================== --- a/drivers/infiniband/core/cma.c 2009-03-10 18:21:47.000000000 +0200 +++ b/drivers/infiniband/core/cma.c 2009-03-10 19:22:18.000000000 +0200 @@ -297,21 +297,25 @@ static void cma_detach_from_dev(struct r id_priv->cma_dev = NULL; } -static int cma_set_qkey(struct ib_device *device, u8 port_num, - enum rdma_port_space ps, - struct rdma_dev_addr *dev_addr, u32 *qkey) +static int cma_set_qkey(struct rdma_id_private *id_priv) { struct ib_sa_mcmember_rec rec; int ret = 0; - switch (ps) { + if (id_priv->qkey) + return; + + switch (id_priv->id.ps) { case RDMA_PS_UDP: - *qkey = RDMA_UDP_QKEY; + id_priv->qkey = RDMA_UDP_QKEY; break; case RDMA_PS_IPOIB: - ib_addr_get_mgid(dev_addr, &rec.mgid); - ret = ib_sa_get_mcmember_rec(device, port_num, &rec.mgid, &rec); - *qkey = be32_to_cpu(rec.qkey); + ib_addr_get_mgid(&id_priv->id.route.addr.dev_addr, &rec.mgid); + ret = ib_sa_get_mcmember_rec(id_priv->id.device, + id_priv->id.port_num, &rec.mgid, + &rec); + if (!ret) + id_priv->qkey = be32_to_cpu(rec.qkey); break; default: break; @@ -341,12 +345,7 @@ static int cma_acquire_dev(struct rdma_i ret = ib_find_cached_gid(cma_dev->device, &gid, &id_priv->id.port_num, NULL); if (!ret) { - ret = cma_set_qkey(cma_dev->device, - id_priv->id.port_num, - id_priv->id.ps, dev_addr, - &id_priv->qkey); - if (!ret) - cma_attach_to_dev(id_priv, cma_dev); + cma_attach_to_dev(id_priv, cma_dev); break; } } @@ -578,6 +577,10 @@ static int cma_ib_init_qp_attr(struct rd *qp_attr_mask = IB_QP_STATE | IB_QP_PKEY_INDEX | IB_QP_PORT; if (cma_is_ud_ps(id_priv->id.ps)) { + ret = cma_set_qkey(id_priv); + if (ret) + return ret; + qp_attr->qkey = id_priv->qkey; *qp_attr_mask |= IB_QP_QKEY; } else { @@ -2201,6 +2204,12 @@ static int cma_sidr_rep_handler(struct i event.status = ib_event->param.sidr_rep_rcvd.status; break; } + ret = cma_set_qkey(id_priv); + if (ret) { + event.event = RDMA_CM_EVENT_ADDR_ERROR; + event.status = -EINVAL; + break; + } if (id_priv->qkey != rep->qkey) { event.event = RDMA_CM_EVENT_UNREACHABLE; event.status = -EINVAL; @@ -2480,10 +2489,14 @@ static int cma_send_sidr_rep(struct rdma const void *private_data, int private_data_len) { struct ib_cm_sidr_rep_param rep; + int ret; memset(&rep, 0, sizeof rep); rep.status = status; if (status == IB_SIDR_SUCCESS) { + ret = cma_set_qkey(id_priv); + if (ret) + return ret; rep.qp_num = id_priv->qp_num; rep.qkey = id_priv->qkey; } From balaji at mcs.anl.gov Wed Mar 25 13:29:23 2009 From: balaji at mcs.anl.gov (Pavan Balaji) Date: Wed, 25 Mar 2009 15:29:23 -0500 Subject: [ofa-general] [hpc-announce] CFP: International Workshop on High Performance Interconnects for Distributed Computing (HPI-DC) Message-ID: <49CA9423.10400@mcs.anl.gov> Workshop on High Performance Interconnects for Distributed Computing (HPI-DC'09) in conjunction with Cluster 2009 August 31, 2009, New Orleans, Louisiana http://www.cercs.gatech.edu/hpidc2009 ********************************************************************* Call for Papers ********************************************************************* The emergence of 10.0 GigE and above, InfiniBand and other high-performance interconnection technologies, programmable NICs and networking platforms, and protocols like DDP and RDMA over IP, make it possible to create tightly linked systems across physical distances that exceed those of traditional single cluster or server systems. These technologies can deliver communication capabilities that achieve the performance levels needed by high end applications in enterprise systems and like those produced by the high performance computing community. Furthermore, the manycore nature of next generation platforms and the creation of distributed cloud computing infrastructure will greatly increase the demand for high performance communication capabilities over wide area distances. The purpose of this workshop is to explore the confluence of distributed computing and communications technologies with high performance interconnects, as applicable or applied to realistic high end applications. The intent is to create a venue that will act as a bridge between researchers developing tools and platforms for high-performance distributed computing, end user applications seeking high performance solutions, and technology providers aiming to improve interconnect and networking technologies for future systems. The hope is to foster knowledge creation and intellectual interchanges between HPC and Cloud computing end users and technology developers, in the specific domain of high performance distributed interconnects. Topics of interest include but are not limited to: # Hardware/software architectures for communication infrastructures for HPC and Cloud Computing # Data and control protocols for interactive and large data volume applications # Novel devices and technologies to enhance interconnect properties # Interconnect-level issues when extending high performance beyond single machines, including architecture, protocols, services, QoS, and security # Remote storage (like iSCSI), remote databases, and datacenters, etc. # Development tools, programming environments and models (like PGAS, OpenShmem, Hadoop, etc.), ranging from programming language support to simulation environments. PAPER SUBMISSIONS: HPI-DC invites authors to submit original and unpublished work. Please submit extended abstracts or full papers, not exceeding 8 double-column pages in 10 point font or larger, in IEEE format. Electronic submission is strongly encouraged. Hard copies will be accepted only if electronic submission is not possible. Submission implies the willingness of at least one of the authors to register and present the paper. Any questions concerning hardcopy submissions or any other issues may be directed to the Program Co-Chairs. IMPORTANT DATES: # Paper submission: June 5th, 2009 # Notification of acceptance: July 10th, 2009 # Final manuscript due: July 29th, 2009 # Workshop date: Aug. 31st, 2009 ORGANIZATION: General Chair # Steve Poole, Oak Ridge National Lab Program Co-Chairs # Pavan Balaji, Argonne National Lab # Ada Gavrilovska, Georgia Institute of Technology Technical Program Committee: # Ahmad Afsahi, Queen's University, Canada # Taisuke Boku, University of Tsukuba, Japan # Ron Brightwell, Sandia National Laboratory # Patrick Geoffray, Myricom # Kei Hiraki, University of Tokyo, Japan # Hyun-wook Jin, Konkuk University, Korea # Pankaj Mehra, HP Research # Guillaume Mercier, INRIA, France # Scott Pakin, Los Alamos National Laboratory # D. K. Panda, Ohio State University # Fabrizio Petrini, IBM Research # Karsten Schwan, Georgia Tech # Jesper Traeff, NEC, Europe # Sudhakar Yalamanchili, Georgia Tech # Weikuan Yu, Auburn University If you have any questions about the workshop, please contact us at hpidc09-chairs at mcs.anl.gov. ==================================================================== If you do not wish to receive any more emails on this list, you can unsubscribe here: https://lists.mcs.anl.gov/mailman/listinfo/hpc-announce ==================================================================== -- Pavan Balaji http://www.mcs.anl.gov/~balaji From YJia at tmriusa.com Wed Mar 25 14:40:08 2009 From: YJia at tmriusa.com (Yicheng Jia) Date: Wed, 25 Mar 2009 16:40:08 -0500 Subject: [ofa-general] OpenSM initialization error Message-ID: Hello, I run into an initialization error during OpenSM start up. The error continues for nearly 1 minutes before the subnet is up. In the log file, i t reports that "__osm_ucast_mgr_process_port: ERR 3A08: No path to get to LID 1 from switch 0x66a00d90008bb". I am using Mellanox MHES18 HCA and Cisco SFS7000P switch. Do you know what the cause is and how to avoid it? Thanks! Yicheng Jia _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From phillipwils at gmail.com Wed Mar 25 19:39:57 2009 From: phillipwils at gmail.com (Phillip Wilson) Date: Wed, 25 Mar 2009 19:39:57 -0700 Subject: [ofa-general] ***SPAM*** System crashed while booting Linux (ia64) with three Mellanox HCAs (15b3:6274) Message-ID: <6e4f44220903251939i68cb4268w1284bbb0552643e7@mail.gmail.com> System crashes with three Mellanox mezzanine cards (VID=15b3, DID=0x6274) installed when booting Linux (ia64). I am using Linux 2.6.24, but this issue also occurs with Linux kernel 2.6.29-rc8. When I added several printk() to the driver function ib_mad_post_receive_mads(), the system was able to boot which implies there is a timing issue in the driver. The system boots if printk() debug messages are added to the ib_mad_post_receive_mads() function within the "../drivers/infiniband/core/mad.c". I have copied the ib_mad_post_receive_mads() function, with my printk to this email. Only debug messages "ib_mad_post_receive_mads(0)" through "ib_mad_post_receive_mads(6)" are seen while booting to Linux. A partial listing from ib_mad_post_receive_mad.S is posted below the "C" code. The exact instruction that cause the system crash was located at ib_mad_post_*+0x0080 st4 [r2]=r3 MII nop.i 0x0 nop.i 0x0 It tries to store r3=0x1600 to [r2] @ 0xE0000007E01C7CCC. <> ib_mthca: Mellanox InfiniBand HCA driver v0.08 (February 14, 2006) ib_mthca: Initializing 0000:05:00.0 GSI 26 (level, low) -> CPU 14 (0x1600) vector 74 ACPI: PCI Interrupt 0000:05:00.0[A] -> GSI 26 (level, low) -> IRQ 74 ib_mthca: Initializing 0000:06:00.0 GSI 30 (level, low) -> CPU 15 (0x1700) vector 78 ACPI: PCI Interrupt 0000:06:00.0[A] -> GSI 30 (level, low) -> IRQ 78 [0] Entered SalCheckHandler [0] SalCheckHandler: Saved Entry State [0] SalCheckHandler: MCA Monarch: 0 [0] SalCheckHandler: Get Sm4 I am using OFED 1.2.5 in user space. HCA firmware on all 3 cards is version 1.2.0 #> lspci -d 15b3: 0000:05:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20) 0000:06:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20) 0000:07:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20) #> uname -a Linux (none) 2.6.24.02.02.08 #3 SMP Wed Mar 25 16:40:52 PST 2009 ia64 unknown /* * Allocate receive MADs and post receive WRs for them */ static int ib_mad_post_receive_mads(struct ib_mad_qp_info *qp_info, struct ib_mad_private *mad) { unsigned long flags; int post, ret; struct ib_mad_private *mad_priv; struct ib_sge sg_list; struct ib_recv_wr recv_wr, *bad_recv_wr; struct ib_mad_queue *recv_queue = &qp_info->recv_queue; printk("ib_mad_post_receive_mads(0)\n"); /* Initialize common scatter list fields */ sg_list.length = sizeof *mad_priv - sizeof mad_priv->header; sg_list.lkey = (*qp_info->port_priv->mr).lkey; /* Initialize common receive WR fields */ recv_wr.next = NULL; recv_wr.sg_list = &sg_list; recv_wr.num_sge = 1; printk("ib_mad_post_receive_mads(1)\n"); do { /* Allocate and map receive buffer */ if (mad) { mad_priv = mad; mad = NULL; } else { printk("ib_mad_post_receive_mads(2)\n"); mad_priv = kmem_cache_alloc(ib_mad_cache, GFP_KERNEL); if (!mad_priv) { printk(KERN_ERR PFX "No memory for receive buffer\n"); ret = -ENOMEM; break; } } printk("ib_mad_post_receive_mads(3)\n"); sg_list.addr = ib_dma_map_single(qp_info->port_priv->device, &mad_priv->grh, sizeof *mad_priv - sizeof mad_priv->header, DMA_FROM_DEVICE); mad_priv->header.mapping = sg_list.addr; recv_wr.wr_id = (unsigned long)&mad_priv->header.mad_list; mad_priv->header.mad_list.mad_queue = recv_queue; /* Post receive WR */ printk("ib_mad_post_receive_mads(4)\n"); spin_lock_irqsave(&recv_queue->lock, flags); post = (++recv_queue->count < recv_queue->max_active); list_add_tail(&mad_priv->header.mad_list.list, &recv_queue->list ); printk("ib_mad_post_receive_mads(5)\n"); spin_unlock_irqrestore(&recv_queue->lock, flags); printk("ib_mad_post_receive_mads(6)\n"); ret = ib_post_recv(qp_info->qp, &recv_wr, &bad_recv_wr); if (ret) { printk("ib_mad_post_receive_mads(7)\n"); spin_lock_irqsave(&recv_queue->lock, flags); printk("ib_mad_post_receive_mads(8)\n"); list_del(&mad_priv->header.mad_list.list); recv_queue->count--; printk("ib_mad_post_receive_mads(9)\n"); spin_unlock_irqrestore(&recv_queue->lock, flags); printk("ib_mad_post_receive_mads(10)\n"); ib_dma_unmap_single(qp_info->port_priv->device, mad_priv->header.mapping, sizeof *mad_priv - sizeof mad_priv->header, DMA_FROM_DEVICE); printk("ib_mad_post_receive_mads(11)\n"); kmem_cache_free(ib_mad_cache, mad_priv); printk("ib_mad_post_receive_mads(12)\n"); printk(KERN_ERR PFX "ib_post_recv failed: %d\n", ret); break; } } while (post); return ret; } "ib_mad_post_receive_mad.S" ib_mad_post_*+0x0000 alloc r50=ar64,0,20,4,0 MII adds r12=-64,r12 mov r18=0x128 ib_mad_post_*+0x0010 mov r10=1;; MMI ld8 r17=[r32] adds r15=24,r12 ib_mad_post_*+0x0020 adds r45=32,r12 MII adds r8=16,r12;; adds r2=16,r45 ib_mad_post_*+0x0030 adds r9=56,r12 MMI mov r51=r1 adds r35=56,r32 ib_mad_post_*+0x0040 adds r14=48,r17 MMI st4 [r15]=r18 mov r49=b0;; ib_mad_post_*+0x0050 ld8 r16=[r14];; MMI adds r11=24,r16 nop.i 0x0;; ib_mad_post_*+0x0060 ld4 r3=[r11] MMI st8 [r2]=r8 adds r2=28,r12 ib_mad_post_*+0x0070 st4 [r9]=r10 MMI st8 [r45]=r0 nop.i 0x0;; ib_mad_post_*+0x0080 st4 [r2]=r3 MII nop.i 0x0 nop.i 0x0 ib_mad_post_*+0x0090 cmp.eq p8,p9=0,r33 MFB nop.f 0x0 nop.b 0x0;; From sfr at canb.auug.org.au Wed Mar 25 20:27:04 2009 From: sfr at canb.auug.org.au (Stephen Rothwell) Date: Thu, 26 Mar 2009 14:27:04 +1100 Subject: [ofa-general] linux-next: manual merge of the net tree with the infiniband tree Message-ID: <20090326142704.20ff442c.sfr@canb.auug.org.au> Hi Dave, Roland, Today's linux-next merge of the net tree got a conflict in drivers/infiniband/hw/cxgb3/cxio_hal.h between commit d1fbe04eee32ed2642cff139b8592866f1d43f41 ("") from the infiniband tree and commit a73efd0a8552927ebe5dff84936f7fdac4f7e314 ("iw_cxgb3: handle chip reset notifications") from the net tree. Just overlapping additions. I fixed it up (see below) and can carry the fix as necessary. -- Cheers, Stephen Rothwell sfr at canb.auug.org.au http://www.canb.auug.org.au/~sfr/ diff --cc drivers/infiniband/hw/cxgb3/cxio_hal.h index e44dc22,9ed65b0..0000000 --- a/drivers/infiniband/hw/cxgb3/cxio_hal.h +++ b/drivers/infiniband/hw/cxgb3/cxio_hal.h @@@ -110,7 -108,8 +110,9 @@@ struct cxio_rdev struct gen_pool *pbl_pool; struct gen_pool *rqt_pool; struct list_head entry; + struct ch_embedded_info fw_info; + u32 flags; + #define CXIO_ERROR_FATAL 1 }; static inline int cxio_num_stags(struct cxio_rdev *rdev_p) From eli at dev.mellanox.co.il Thu Mar 26 00:51:35 2009 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Thu, 26 Mar 2009 09:51:35 +0200 Subject: [ofa-general] Re: [ewg] Re: [PATCH] IB/ipoib: set neigh->dgid upon ipoib_neigh creation In-Reply-To: <49CA83F3.9040204@Voltaire.COM> References: <20090325145114.GA31206@mtls03> <49CA83F3.9040204@Voltaire.COM> Message-ID: <20090326075135.GA31246@mtls03> On Wed, Mar 25, 2009 at 09:20:19PM +0200, Yossi Etigin wrote: > Wouldn't this patch break the fast bond+SM failover recovery? > > Up till now, in such case neigh->dgid was the old gid until path record > query was successful, and that caused to retry the path query until it's > successful. With that patch, a new path query will not be issued until > the next arp refresh because neigh->dgid will be set to a valid value > right after the first packet. Looks to me like > How about keeping the memcpy in path_rec-completion, and also adding it > to ib_mcast_send (where ipoib neigh is created)? > I did not like the idea that a unicast ipoib_neigh can be created and destroyed till the path is resolved so I preferred to initialize at creation time. How about the following patch to solve the problem of failure to resolve a path: >From 4e6e3dd0186803f7547f5e4c233d7525dfcdaec6 Mon Sep 17 00:00:00 2001 From: Eli Cohen Date: Wed, 25 Mar 2009 16:22:28 +0200 Subject: [PATCH] IB/ipoib: set neigh->dgid upon ipoib_neigh creation If we fail to do that, there can be superfluous calls to ipoib_neigh_free()/ipoib_neigh_alloc() due to the following if statement in ipoib_start_xmit(): if (unlikely((memcmp(&neigh->dgid.raw, skb->dst->neighbour->ha + 4, sizeof(union ib_gid))) || In the case of a unicast GID, it can happen until the path is resolved and is not so severe. In the case of a multicast address, neigh->dgid is never even updated, so the corresponding ipoib_neigh will be destroyed and created on every multicast packet sent. Signed-off-by: Eli Cohen --- drivers/infiniband/ulp/ipoib/ipoib_main.c | 7 ++++--- 1 files changed, 4 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c index 0bd2a4f..b680483 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c @@ -460,8 +460,6 @@ static void path_rec_completion(int status, } kref_get(&path->ah->ref); neigh->ah = path->ah; - memcpy(&neigh->dgid.raw, &path->pathrec.dgid.raw, - sizeof(union ib_gid)); if (ipoib_cm_enabled(dev, neigh->neighbour)) { if (!ipoib_cm_get(neigh)) @@ -481,7 +479,9 @@ static void path_rec_completion(int status, __skb_queue_tail(&skqueue, skb); } path->valid = 1; - } + } else + list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) + memset(&neigh->dgid, 0, sizeof neigh->dgid); path->query = NULL; complete(&path->done); @@ -878,6 +878,7 @@ struct ipoib_neigh *ipoib_neigh_alloc(struct neighbour *neighbour, neigh->neighbour = neighbour; neigh->dev = dev; *to_ipoib_neigh(neighbour) = neigh; + memcpy(neigh->dgid.raw, neighbour->ha + 4, 16); skb_queue_head_init(&neigh->queue); ipoib_cm_set(neigh, NULL); -- 1.6.2.1 From ogerlitz at voltaire.com Thu Mar 26 01:52:33 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 26 Mar 2009 10:52:33 +0200 (IST) Subject: [ofa-general] [PATCH] doc/ipoib: document CM, offloads, interrupt moderation Message-ID: Update the documentation to include connected mode, stateless offloads and interrupt moderation. Signed-off-by: Or Gerlitz Index: linus-linux-2.6/Documentation/infiniband/ipoib.txt =================================================================== --- linus-linux-2.6.orig/Documentation/infiniband/ipoib.txt +++ linus-linux-2.6/Documentation/infiniband/ipoib.txt @@ -24,6 +24,46 @@ Partitions and P_Keys The P_Key for any interface is given by the "pkey" file, and the main interface for a subinterface is in "parent." +Datagram vs Connected modes + + The driver supports two modes of operation - datagram and connected. + The mode is set/read through the /sys/class/net/ibXXX/mode device entry. + + For datagram mode, the IB UD (Unrelaible Datagram) transport is used + and as such the device MTU has very close proximity to the IB L2 MTU + which is typically 2k (larger MTU of 4K supported by the code). + + For connected mode, the IB RC (Reliable Connection) transport is used. + The main reasoning behind connected mode is to take advantage of the + connected nature of the IB transport and use MTUs up to the maximal + IP packet size of 64K - which reduces the number of IP packets needed for + handling large UDP datagrams, TCP segments, etc and increases the performance. + + Under connected mode, the device UD QP is still used for multicast and + communication with peers who don't support connected mode. In this case, + RX emulation of ICMP PMTU packets is used for having the stack use + the smaller UD MTU for these neighbours. + +Stateless offloads + + In case the underlying IB HW supports that, IPoIB advertizes TCP checksum + and/or Large Send (LSO) offloading capability to the network stack. This + is done by the device setting the relevant bits on its features bitmask. + + Large Receive (LRO) offloading is also implemented and may be turned on/off + using ethtool calls, for the time being its supported only for checksum + offload capable devices. + + Offloads are supported only under datagram mode. + +Interrupt moderation + + In case the underlying IB device supports CQ event moderation, one can + have run time control on the rate of interrupt being generated and thus + reduce the overhead incurred by handling interrupts. For the most case + IPoIB doesn't use events for TX completion signaling and as such, only + RX moderation is supported. + Debugging Information By compiling the IPoIB driver with CONFIG_INFINIBAND_IPOIB_DEBUG set @@ -55,3 +95,5 @@ References http://ietf.org/rfc/rfc4391.txt IP over InfiniBand (IPoIB) Architecture (RFC 4392) http://ietf.org/rfc/rfc4392.txt + IP over InfiniBand: Connected Mode (RFC 4755) + http://ietf.org/rfc/rfc4755.txt From vlad at lists.openfabrics.org Thu Mar 26 03:33:22 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Thu, 26 Mar 2009 03:33:22 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090326-0200 daily build status Message-ID: <20090326103322.64A99E61019@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From nicolas.morey-chaisemartin at ext.bull.net Thu Mar 26 05:37:55 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Thu, 26 Mar 2009 13:37:55 +0100 Subject: [ofa-general] OpenSM and trap 128. Message-ID: <49CB7723.9080104@ext.bull.net> Hi, We've noticed while setting up a new cluster a problem with OpenSM. As usual, there are some cable problems while plugging the cluster but one of the cable was changing state over 10 000 thousands times per second (OFF/ON) and sending each time a 128 trap to OpenSM. Therefore, OpenSM is constantly rerouting the whole interconnect (every 10s or so). Fixing the cable will solve our problem, but I still think something should be done about this. Though OpenSM behaviour was OK, it was really difficult to find where the performances problems came from. All our diagnostics tools (mostly using infiniband diags) were failing to see the problem. Infiniband diags commands fail toward the faulty port but it was hard to say if port was faulty or if it was due to high load on the SM and dropped VL15 messages. I was thinking about a solution: When receiving a 128 trap (and it triggers a heavy sweep) we check the faulty GUID, lid or port guid. If last heavy sweep was triggered by the same faulty port, we wait twice last the last waiting time before forcing the new heavy sweep. If it's another source or another reason, we force the heavy sweep right then and set the waiting time to 0. This way, different problem will still trigger a heavy sweep asap but if only one faulty links triggers it it'll sweep less and less often as it is pretty useless. It should solve this case but there may still be a problem when more ports have the same problem... Any idea on a way to manage this? An ignore mask on traps? (ignore traps for 1 specific problem for x seconds if they happen to often) Thanks Nicolas From hal.rosenstock at gmail.com Thu Mar 26 07:48:07 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 26 Mar 2009 10:48:07 -0400 Subject: ***SPAM*** Re: [ofa-general] OpenSM and trap 128. In-Reply-To: <49CB7723.9080104@ext.bull.net> References: <49CB7723.9080104@ext.bull.net> Message-ID: On Thu, Mar 26, 2009 at 8:37 AM, Nicolas Morey Chaisemartin wrote: > Hi, > > We've noticed while setting up a new cluster a problem with OpenSM. > As usual, there are some cable problems while plugging the cluster but one of the cable was changing state over 10 000 thousands times per second (OFF/ON) and sending each time a 128 trap to OpenSM. That's a violation of the IBTA spec by those SMAs. There's a max rate that traps are supposed to be generated at based on the response time. That's even if the actual event rate exceeds that which sounds like the case here. This issue has been known for several years now but still is unfixed :-( > Therefore, OpenSM is constantly rerouting the whole interconnect (every 10s or so). Even if the trap rate issue is resolved (chewing up lots of SM CPU), it won't do anything about this issue. > Fixing the cable will solve our problem, but I still think something should be done about this. > > Though OpenSM behaviour was OK, it was really difficult to find where the performances problems came from. There should be some log messages as to the trap rate being exceeded. Were they not present ? Which OpenSM version ? > All our diagnostics tools (mostly using infiniband diags) were failing to see the problem. > Infiniband diags commands fail toward the faulty port but it was hard to say if port was faulty or if it was due to high load on the SM and dropped VL15 messages. Yes, the only thing you would observe is VL15 drops via perfquery. The SM is the one which should be logging the trap originator which is the way to diagnose this issue. > I was thinking about a solution: > When receiving a 128 trap (and it triggers a heavy sweep) we check the faulty GUID, lid or port guid. > If last heavy sweep was triggered by the same faulty port, we wait twice last the last waiting time before forcing the new heavy sweep. > If it's another source or another reason, we force the heavy sweep right then and set the waiting time to 0. > > This way, different problem will still trigger a heavy sweep asap but if only one faulty links triggers it it'll sweep less and less often as it is pretty useless. > > It should solve this case but there may still be a problem when more ports have the same problem... > > Any idea on a way to manage this? > An ignore mask on traps? (ignore traps for 1 specific problem for x seconds if they happen to often) I need to think about this more before commenting. -- Hal > Thanks > > Nicolas > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From chas at cmf.nrl.navy.mil Thu Mar 26 07:56:08 2009 From: chas at cmf.nrl.navy.mil (Chas Williams (CONTRACTOR)) Date: Thu, 26 Mar 2009 10:56:08 -0400 Subject: [ofa-general] OpenSM and trap 128. In-Reply-To: <49CB7723.9080104@ext.bull.net> Message-ID: <200903261456.n2QEu8xI014729@cmf.nrl.navy.mil> In message <49CB7723.9080104 at ext.bull.net>,Nicolas Morey Chaisemartin writes: >We've noticed while setting up a new cluster a problem with OpenSM. >As usual, there are some cable problems while plugging the cluster but one of >the cable was changing state over 10 000 thousands times per second (OFF/ON) a >nd sending each time a 128 trap to OpenSM. >Therefore, OpenSM is constantly rerouting the whole interconnect (every 10s or > so). >Fixing the cable will solve our problem, but I still think something should be > done about this. we have seen the same problem here locally. it seems to be a violation of the spec to send this many traps per second. >I was thinking about a solution: >When receiving a 128 trap (and it triggers a heavy sweep) we check the faulty >GUID, lid or port guid. >If last heavy sweep was triggered by the same faulty port, we wait twice last >the last waiting time before forcing the new heavy sweep. >If it's another source or another reason, we force the heavy sweep right then >and set the waiting time to 0. > >This way, different problem will still trigger a heavy sweep asap but if only >one faulty links triggers it it'll sweep less and less often as it is pretty u >seless. > >It should solve this case but there may still be a problem when more ports hav >e the same problem... > >Any idea on a way to manage this? >An ignore mask on traps? (ignore traps for 1 specific problem for x seconds if > they happen to often) our solution was a custom patch (that might have made it into the opensm distribution) called 'babbling_port_policy'. it attempted to disable the port in question. From hal.rosenstock at gmail.com Thu Mar 26 07:58:50 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 26 Mar 2009 10:58:50 -0400 Subject: ***SPAM*** Re: [ofa-general] OpenSM initialization error In-Reply-To: References: Message-ID: 2009/3/25 Yicheng Jia : > > Hello, > > I run into an initialization error during OpenSM start up. The error > continues for nearly 1 minutes before the subnet is up. In the log file, it > reports that "__osm_ucast_mgr_process_port: ERR 3A08: No path to get to LID > 1 from switch 0x66a00d90008bb". > > I am using Mellanox MHES18 HCA and Cisco SFS7000P switch. Do you know what > the cause is and how to avoid it? What is your topology ? Does this message persist or go away in the log ? Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. Amongst other changes, this message has been downgraded from error to debug in more recent versions. -- Hal > > Thanks! > > Yicheng Jia > > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From YJia at tmriusa.com Thu Mar 26 08:16:02 2009 From: YJia at tmriusa.com (Yicheng Jia) Date: Thu, 26 Mar 2009 10:16:02 -0500 Subject: [ofa-general] OpenSM initialization error In-Reply-To: Message-ID: Hi Hal, > What is your topology ? Several nodes directly connect to the switch. > Does this message persist or go away in the log ? It continues reporting this message until the subnet is up, then this message goes away. It lasts about 1 minute. > Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. > Amongst other changes, this message has been downgraded from error to > debug in more recent versions. Do you know what could cause the problem? Is there any way to make the error time shorter? Thanks! Yicheng Jia Hal Rosenstock 03/26/2009 09:58 AM To Yicheng Jia cc hnrose at comcast.net, general at lists.openfabrics.org Subject Re: [ofa-general] OpenSM initialization error 2009/3/25 Yicheng Jia : > > Hello, > > I run into an initialization error during OpenSM start up. The error > continues for nearly 1 minutes before the subnet is up. In the log file, it > reports that "__osm_ucast_mgr_process_port: ERR 3A08: No path to get to LID > 1 from switch 0x66a00d90008bb". > > I am using Mellanox MHES18 HCA and Cisco SFS7000P switch. Do you know what > the cause is and how to avoid it? What is your topology ? Does this message persist or go away in the log ? Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. Amongst other changes, this message has been downgraded from error to debug in more recent versions. -- Hal > > Thanks! > > Yicheng Jia > > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From nicolas.morey-chaisemartin at ext.bull.net Thu Mar 26 08:20:57 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Thu, 26 Mar 2009 16:20:57 +0100 Subject: [ofa-general] OpenSM and trap 128. In-Reply-To: References: <49CB7723.9080104@ext.bull.net> Message-ID: <49CB9D59.9050301@ext.bull.net> Hal Rosenstock wrote: > >> Fixing the cable will solve our problem, but I still think something should be done about this. >> >> Though OpenSM behaviour was OK, it was really difficult to find where the performances problems came from. > > There should be some log messages as to the trap rate being exceeded. > Were they not present ? Which OpenSM version ? Only message we had are the events on trap reception (so a real lots of them). However we didn't check that before spending quite some time trying to understand where performances loss could come from. OpenSM is git head + Bull_patches on the top. > >> All our diagnostics tools (mostly using infiniband diags) were failing to see the problem. >> Infiniband diags commands fail toward the faulty port but it was hard to say if port was faulty or if it was due to high load on the SM and dropped VL15 messages. > > Yes, the only thing you would observe is VL15 drops via perfquery. The > SM is the one which should be logging the trap originator which is the > way to diagnose this issue. > It is actually. Though it's missing the port number in the log message. Nicolas From hal.rosenstock at gmail.com Thu Mar 26 08:24:35 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 26 Mar 2009 11:24:35 -0400 Subject: ***SPAM*** Re: [ofa-general] OpenSM and trap 128. In-Reply-To: <200903261456.n2QEu8xI014729@cmf.nrl.navy.mil> References: <49CB7723.9080104@ext.bull.net> <200903261456.n2QEu8xI014729@cmf.nrl.navy.mil> Message-ID: On Thu, Mar 26, 2009 at 10:56 AM, Chas Williams (CONTRACTOR) wrote: > our solution was a custom patch (that might have made it into the opensm > distribution) called 'babbling_port_policy'.  it attempted to disable the > port in question. It's part of the "normal" OpenSM for quite some time now. Turning on babbling_port_policy will help the SM side but I think the rerouting will still occur as the rate of traps is still more than 1 every 10 seconds. -- Hal From hal.rosenstock at gmail.com Thu Mar 26 08:28:07 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 26 Mar 2009 11:28:07 -0400 Subject: ***SPAM*** Re: [ofa-general] OpenSM and trap 128. In-Reply-To: <49CB9D59.9050301@ext.bull.net> References: <49CB7723.9080104@ext.bull.net> <49CB9D59.9050301@ext.bull.net> Message-ID: On Thu, Mar 26, 2009 at 11:20 AM, Nicolas Morey Chaisemartin wrote: > Hal Rosenstock wrote: >> >>> Fixing the cable will solve our problem, but I still think something should be done about this. >>> >>> Though OpenSM behaviour was OK, it was really difficult to find where the performances problems came from. >> >> There should be some log messages as to the trap rate being exceeded. >> Were they not present ? Which OpenSM version ? > > Only message we had are the events on trap reception (so a real lots of them). > However we didn't check that before spending quite some time trying to understand where performances loss could come from. > OpenSM is git head + Bull_patches on the top. So babbling_port_policy should be there. >> >>> All our diagnostics tools (mostly using infiniband diags) were failing to see the problem. >>> Infiniband diags commands fail toward the faulty port but it was hard to say if port was faulty or if it was due to high load on the SM and dropped VL15 messages. >> >> Yes, the only thing you would observe is VL15 drops via perfquery. The >> SM is the one which should be logging the trap originator which is the >> way to diagnose this issue. >> > > It is actually. Though it's missing the port number in the log message. The trap itself does not contain the port number so the log message can't contain it. The reason is simplicity as multiple ports may have this condition. -- Hal > Nicolas > From hal.rosenstock at gmail.com Thu Mar 26 08:32:42 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 26 Mar 2009 11:32:42 -0400 Subject: ***SPAM*** Re: [ofa-general] OpenSM initialization error In-Reply-To: References: Message-ID: On Thu, Mar 26, 2009 at 11:16 AM, Yicheng Jia wrote: > > Hi Hal, > >> What is your topology ? > > Several nodes directly connect to the switch. > >> Does this message persist or go away in the log ? > > It continues reporting this message until the subnet is up, then this > message goes away. It lasts about 1 minute. What is LID 1 ? Is it the switch ? Is Cisco SM running ? >> Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. >> Amongst other changes, this message has been downgraded from error to >> debug in more recent versions. > > Do you know what could cause the problem? Is there any way to make the error > time shorter? What's the exact scenario under which this occurs ? -- Hal > Thanks! > > Yicheng Jia > > > > > Hal Rosenstock > > 03/26/2009 09:58 AM > > To > Yicheng Jia > cc > hnrose at comcast.net, general at lists.openfabrics.org > Subject > Re: [ofa-general] OpenSM initialization error > > > > > 2009/3/25 Yicheng Jia : >> >> Hello, >> >> I run into an initialization error during OpenSM start up. The error >> continues for nearly 1 minutes before the subnet is up. In the log file, >> it >> reports that "__osm_ucast_mgr_process_port: ERR 3A08: No path to get to >> LID >> 1 from switch 0x66a00d90008bb". >> >> I am using Mellanox MHES18 HCA and Cisco SFS7000P switch. Do you know what >> the cause is and how to avoid it? > > What is your topology ? > > Does this message persist or go away in the log ? > > Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. > Amongst other changes, this message has been downgraded from error to > debug in more recent versions. > > -- Hal > >> >> Thanks! >> >> Yicheng Jia >> >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> >> _____________________________________________________________________________ >> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > From hnrose at comcast.net Thu Mar 26 09:39:57 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 26 Mar 2009 11:39:57 -0500 Subject: [ofa-general] ***SPAM*** [PATCH][TRIVIAL] opensm/osm_ucast_mgr.c: Cosmetic formatting change Message-ID: <20090326163956.GA31271@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c index 7896fe5..a194ab0 100644 --- a/opensm/opensm/osm_ucast_mgr.c +++ b/opensm/opensm/osm_ucast_mgr.c @@ -270,8 +270,7 @@ static void ucast_mgr_process_port(IN osm_ucast_mgr_t * p_mgr, osm_physp_t *p = osm_node_get_physp_ptr(p_sw->p_node, port); OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG, - "Routing LID %u to port %u" - " for switch 0x%" PRIx64 "\n", + "Routing LID %u to port %u for switch 0x%" PRIx64 "\n", lid_ho, port, cl_ntoh64(node_guid)); /* From hnrose at comcast.net Thu Mar 26 09:39:08 2009 From: hnrose at comcast.net (Hal Rosenstock) Date: Thu, 26 Mar 2009 11:39:08 -0500 Subject: [ofa-general] ***SPAM*** [PATCH] libibmad/fields.c: Display CounterSelect2 in hex rather than decimal Message-ID: <20090326163908.GA31265@comcast.net> Signed-off-by: Hal Rosenstock --- diff --git a/libibmad/src/fields.c b/libibmad/src/fields.c index 19c8fc1..df43ceb 100644 --- a/libibmad/src/fields.c +++ b/libibmad/src/fields.c @@ -251,7 +251,7 @@ static const ib_field_t ib_mad_f[] = { {BITSOFFS(112, 16), "XmtDiscards", mad_dump_uint}, {BITSOFFS(128, 8), "XmtConstraintErrors", mad_dump_uint}, {BITSOFFS(136, 8), "RcvConstraintErrors", mad_dump_uint}, - {BITSOFFS(144, 8), "CounterSelect2", mad_dump_uint}, + {BITSOFFS(144, 8), "CounterSelect2", mad_dump_hex}, {BITSOFFS(152, 4), "LinkIntegrityErrors", mad_dump_uint}, {BITSOFFS(156, 4), "ExcBufOverrunErrors", mad_dump_uint}, {BITSOFFS(176, 16), "VL15Dropped", mad_dump_uint}, From rdreier at cisco.com Thu Mar 26 08:54:16 2009 From: rdreier at cisco.com (Roland Dreier) Date: Thu, 26 Mar 2009 08:54:16 -0700 Subject: [ofa-general] Re: System crashed while booting Linux (ia64) with three Mellanox HCAs (15b3:6274) In-Reply-To: <6e4f44220903251939i68cb4268w1284bbb0552643e7@mail.gmail.com> (Phillip Wilson's message of "Wed, 25 Mar 2009 19:39:57 -0700") References: <6e4f44220903251939i68cb4268w1284bbb0552643e7@mail.gmail.com> Message-ID: > System crashes with three Mellanox mezzanine cards (VID=15b3, > DID=0x6274) installed when booting Linux (ia64). I am using Linux > 2.6.24, but this issue also occurs with Linux kernel 2.6.29-rc8. this is a pretty interesting crash. Do you have the ib_mthca driver built into your kernel, or is it being loaded as a module? > A partial listing from ib_mad_post_receive_mad.S is posted below the "C" code. > The exact instruction that cause the system crash was located at > > ib_mad_post_*+0x0080 st4 [r2]=r3 MII > nop.i 0x0 > nop.i 0x0 > > It tries to store r3=0x1600 to [r2] @ 0xE0000007E01C7CCC. Looking at the assembly, it seems the relevant parts are: ib_mad_post_*+0x0060 ld4 r3=[r11] MMI st8 [r2]=r8 adds r2=28,r12 ib_mad_post_*+0x0070 st4 [r9]=r10 MMI st8 [r45]=r0 nop.i 0x0;; ib_mad_post_*+0x0080 st4 [r2]=r3 MII The main points are "adds r2=28,r12" -- ie r2 now points into the stack -- and "st4 [r2]=r3" -- ie a store onto the stack is crashing. In the same function, we have "adds r9=56,r12" and "st4 [r9]=r10" slightly earlier, so the stack isn't totally messed up (apparently). Not sure how to debug this since the crash as it stands doesn't seem to make much sense... From YJia at tmriusa.com Thu Mar 26 09:01:05 2009 From: YJia at tmriusa.com (Yicheng Jia) Date: Thu, 26 Mar 2009 11:01:05 -0500 Subject: [ofa-general] OpenSM initialization error In-Reply-To: Message-ID: LID 1 is the node that OpenSM runs on. Cisco SFS7000P is managed switch, SM is running all the time. This occurs during the system starts up. At first Mthca driver 1.3 starts on all nodes, after the driver is up, the OpenSM starts on one node, then the error shows up. It also occurs on Qlogic unmanaged switch. Is this related to hardware? Thanks! Yicheng Jia Hal Rosenstock 03/26/2009 10:32 AM To Yicheng Jia cc general at lists.openfabrics.org, hnrose at comcast.net Subject Re: [ofa-general] OpenSM initialization error On Thu, Mar 26, 2009 at 11:16 AM, Yicheng Jia wrote: > > Hi Hal, > >> What is your topology ? > > Several nodes directly connect to the switch. > >> Does this message persist or go away in the log ? > > It continues reporting this message until the subnet is up, then this > message goes away. It lasts about 1 minute. What is LID 1 ? Is it the switch ? Is Cisco SM running ? >> Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. >> Amongst other changes, this message has been downgraded from error to >> debug in more recent versions. > > Do you know what could cause the problem? Is there any way to make the error > time shorter? What's the exact scenario under which this occurs ? -- Hal > Thanks! > > Yicheng Jia > > > > > Hal Rosenstock > > 03/26/2009 09:58 AM > > To > Yicheng Jia > cc > hnrose at comcast.net, general at lists.openfabrics.org > Subject > Re: [ofa-general] OpenSM initialization error > > > > > 2009/3/25 Yicheng Jia : >> >> Hello, >> >> I run into an initialization error during OpenSM start up. The error >> continues for nearly 1 minutes before the subnet is up. In the log file, >> it >> reports that "__osm_ucast_mgr_process_port: ERR 3A08: No path to get to >> LID >> 1 from switch 0x66a00d90008bb". >> >> I am using Mellanox MHES18 HCA and Cisco SFS7000P switch. Do you know what >> the cause is and how to avoid it? > > What is your topology ? > > Does this message persist or go away in the log ? > > Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. > Amongst other changes, this message has been downgraded from error to > debug in more recent versions. > > -- Hal > >> >> Thanks! >> >> Yicheng Jia >> >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> >> _____________________________________________________________________________ >> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yosefe at Voltaire.COM Thu Mar 26 10:05:56 2009 From: yosefe at Voltaire.COM (Yossi Etigin) Date: Thu, 26 Mar 2009 19:05:56 +0200 Subject: [ofa-general] Re: [ewg] Re: [PATCH] IB/ipoib: set neigh->dgid upon ipoib_neigh creation In-Reply-To: <20090326075135.GA31246@mtls03> References: <20090325145114.GA31206@mtls03> <49CA83F3.9040204@Voltaire.COM> <20090326075135.GA31246@mtls03> Message-ID: <49CBB5F4.1080701@Voltaire.COM> Eli Cohen wrote: > @@ -481,7 +479,9 @@ static void path_rec_completion(int status, > __skb_queue_tail(&skqueue, skb); > } > path->valid = 1; > - } > + } else > + list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) > + memset(&neigh->dgid, 0, sizeof neigh->dgid); > > path->query = NULL; > complete(&path->done); In case of SM failover, there is a period of time where there is no sm_ah in sa, so ib_sa_path_rec_get() returns -EAGAIN and path_rec_start() fails. This means the code above will not be called, and again - no path until next arp refresh. --Yossi From hal.rosenstock at gmail.com Thu Mar 26 10:13:50 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 26 Mar 2009 13:13:50 -0400 Subject: ***SPAM*** Re: [ofa-general] OpenSM initialization error In-Reply-To: References: Message-ID: On Thu, Mar 26, 2009 at 12:01 PM, Yicheng Jia wrote: > > LID 1 is the node that OpenSM runs on. Cisco SFS7000P is managed switch, SM > is running all the time. You shouldn't be running different flavor SMs on the same subnet. > This occurs during the system starts up. At first Mthca driver 1.3 starts on > all nodes, after the driver is up, the OpenSM starts on one node, then the > error shows up. It also occurs on Qlogic unmanaged switch. Are you sure there's no SM running here other than OpenSM ? -- Hal > Is this related to hardware? > Thanks! > > Yicheng Jia > > > > > Hal Rosenstock > > 03/26/2009 10:32 AM > > To > Yicheng Jia > cc > general at lists.openfabrics.org, hnrose at comcast.net > Subject > Re: [ofa-general] OpenSM initialization error > > > > > On Thu, Mar 26, 2009 at 11:16 AM, Yicheng Jia wrote: >> >> Hi Hal, >> >>> What is your topology ? >> >> Several nodes directly connect to the switch. >> >>> Does this message persist or go away in the log ? >> >> It continues reporting this message until the subnet is up, then this >> message goes away. It lasts about 1 minute. > > What is LID 1 ? Is it the switch ? Is Cisco SM running ? > >>> Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. >>> Amongst other changes, this message has been downgraded from error to >>> debug in more recent versions. >> >> Do you know what could cause the problem? Is there any way to make the >> error >> time shorter? > > What's the exact scenario under which this occurs ? > > -- Hal > >> Thanks! >> >> Yicheng Jia >> >> >> >> >> Hal Rosenstock >> >> 03/26/2009 09:58 AM >> >> To >> Yicheng Jia >> cc >> hnrose at comcast.net, general at lists.openfabrics.org >> Subject >> Re: [ofa-general] OpenSM initialization error >> >> >> >> >> 2009/3/25 Yicheng Jia : >>> >>> Hello, >>> >>> I run into an initialization error during OpenSM start up. The error >>> continues for nearly 1 minutes before the subnet is up. In the log file, >>> it >>> reports that "__osm_ucast_mgr_process_port: ERR 3A08: No path to get to >>> LID >>> 1 from switch 0x66a00d90008bb". >>> >>> I am using Mellanox MHES18 HCA and Cisco SFS7000P switch. Do you know >>> what >>> the cause is and how to avoid it? >> >> What is your topology ? >> >> Does this message persist or go away in the log ? >> >> Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. >> Amongst other changes, this message has been downgraded from error to >> debug in more recent versions. >> >> -- Hal >> >>> >>> Thanks! >>> >>> Yicheng Jia >>> >>> >>> >>> >>> _____________________________________________________________________________ >>> Scanned by IBM Email Security Management Services powered by MessageLabs. >>> For more information please visit http://www.ers.ibm.com >>> >>> >>> _____________________________________________________________________________ >>> >>> _______________________________________________ >>> general mailing list >>> general at lists.openfabrics.org >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>> >>> To unsubscribe, please visit >>> http://openib.org/mailman/listinfo/openib-general >>> >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> >> _____________________________________________________________________________ >> >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> >> _____________________________________________________________________________ >> > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > From vuhuong at mellanox.com Thu Mar 26 10:21:37 2009 From: vuhuong at mellanox.com (Vu Pham) Date: Thu, 26 Mar 2009 10:21:37 -0700 Subject: [ofa-general] NFS-RDMA server run out-of-memory, Message-ID: <49CBB9A1.9020209@mellanox.com> Jon, nfsrdma client (RHEL 5.2), Arbel or ConnectX HCAs nfsrdma server (RHEL 5.2/5.3 or 2.6.27), Arbel HCA I run connectathon -N 1000 (or less if you have less memory on nfsrdma server), System run out-of-memory and reboot. Do you see the same behavior with Chelsio? Call Trace: [] out_of_memory+0x8e/0x2f5 [] __alloc_pages+0x245/0x2ce [] alloc_page_interleave+0x3d/0x74 [] __do_page_cache_readahead+0x95/0x1d9 [] zone_statistics+0x3e/0x6d [] ifind_fast+0x47/0x83 [] iput+0x4b/0x84 [] blockable_page_cache_readahead+0x53/0xb2 [] page_cache_readahead+0x13d/0x1af [] do_generic_mapping_read+0x126/0x3f8 [] :nfsd:nfsd_read_actor+0x0/0xd9 [] generic_file_sendfile+0x4c/0x64 [] :nfsd:nfsd_vfs_read+0x20a/0x32f [] :nfsd:nfsd_read+0x9c/0xba [] :nfsd:nfsd3_proc_read+0x11b/0x161 [] :nfsd:nfsd_dispatch+0xde/0x1b6 [] :sunrpc:svc_process+0x405/0x6de [] __down_read+0x12/0x92 [] keventd_create_kthread+0x0/0xc4 [] :nfsd:nfsd+0x1a3/0x274 [] :nfsd:nfsd+0x0/0x274 [] kthread+0xfe/0x132 [] child_rip+0xa/0x11 [] keventd_create_kthread+0x0/0xc4 [] kthread+0x0/0x132 [] child_rip+0x0/0x11 Node 0 DMA per-cpu: cpu 0 hot: high 0, batch 1 used:0 cpu 0 cold: high 0, batch 1 used:0 cpu 1 hot: high 0, batch 1 used:0 cpu 1 cold: high 0, batch 1 used:0 cpu 2 hot: high 0, batch 1 used:0 cpu 2 cold: high 0, batch 1 used:0 cpu 3 hot: high 0, batch 1 used:0 cpu 3 cold: high 0, batch 1 used:0 Node 0 DMA32 per-cpu: Call Trace: [] out_of_memory+0x8e/0x2f5 [] __alloc_pages+0x245/0x2ce [] _spin_lock_bh+0x9/0x14 [] alloc_page_interleave+0x3d/0x74 [] :sunrpc:svc_recv+0xc2/0x73f [] default_wake_function+0x0/0xe [] __down_read+0x12/0x92 [] keventd_create_kthread+0x0/0xc4 [] :nfsd:nfsd+0xd3/0x274 [] :nfsd:nfsd+0x0/0x274 [] kthread+0xfe/0x132 [] child_rip+0xa/0x11 [] keventd_create_kthread+0x0/0xc4 [] kthread+0x0/0x132 [] child_rip+0x0/0x11 -vu From YJia at tmriusa.com Thu Mar 26 10:46:24 2009 From: YJia at tmriusa.com (Yicheng Jia) Date: Thu, 26 Mar 2009 12:46:24 -0500 Subject: [ofa-general] OpenSM initialization error In-Reply-To: Message-ID: > You shouldn't be running different flavor SMs on the same subnet. The OpenSM will go to standby if there's another SM exists. > Are you sure there's no SM running here other than OpenSM ? Actually this error always happens on Qlogic unmanaged switch, which I believe no SM running on the subnet. It always takes more than 1 minute for the error disappear. Thanks! Yicheng Jia Hal Rosenstock 03/26/2009 12:13 PM To Yicheng Jia cc general at lists.openfabrics.org, hnrose at comcast.net Subject Re: [ofa-general] OpenSM initialization error On Thu, Mar 26, 2009 at 12:01 PM, Yicheng Jia wrote: > > LID 1 is the node that OpenSM runs on. Cisco SFS7000P is managed switch, SM > is running all the time. You shouldn't be running different flavor SMs on the same subnet. > This occurs during the system starts up. At first Mthca driver 1.3 starts on > all nodes, after the driver is up, the OpenSM starts on one node, then the > error shows up. It also occurs on Qlogic unmanaged switch. Are you sure there's no SM running here other than OpenSM ? -- Hal > Is this related to hardware? > Thanks! > > Yicheng Jia > > > > > Hal Rosenstock > > 03/26/2009 10:32 AM > > To > Yicheng Jia > cc > general at lists.openfabrics.org, hnrose at comcast.net > Subject > Re: [ofa-general] OpenSM initialization error > > > > > On Thu, Mar 26, 2009 at 11:16 AM, Yicheng Jia wrote: >> >> Hi Hal, >> >>> What is your topology ? >> >> Several nodes directly connect to the switch. >> >>> Does this message persist or go away in the log ? >> >> It continues reporting this message until the subnet is up, then this >> message goes away. It lasts about 1 minute. > > What is LID 1 ? Is it the switch ? Is Cisco SM running ? > >>> Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. >>> Amongst other changes, this message has been downgraded from error to >>> debug in more recent versions. >> >> Do you know what could cause the problem? Is there any way to make the >> error >> time shorter? > > What's the exact scenario under which this occurs ? > > -- Hal > >> Thanks! >> >> Yicheng Jia >> >> >> >> >> Hal Rosenstock >> >> 03/26/2009 09:58 AM >> >> To >> Yicheng Jia >> cc >> hnrose at comcast.net, general at lists.openfabrics.org >> Subject >> Re: [ofa-general] OpenSM initialization error >> >> >> >> >> 2009/3/25 Yicheng Jia : >>> >>> Hello, >>> >>> I run into an initialization error during OpenSM start up. The error >>> continues for nearly 1 minutes before the subnet is up. In the log file, >>> it >>> reports that "__osm_ucast_mgr_process_port: ERR 3A08: No path to get to >>> LID >>> 1 from switch 0x66a00d90008bb". >>> >>> I am using Mellanox MHES18 HCA and Cisco SFS7000P switch. Do you know >>> what >>> the cause is and how to avoid it? >> >> What is your topology ? >> >> Does this message persist or go away in the log ? >> >> Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. >> Amongst other changes, this message has been downgraded from error to >> debug in more recent versions. >> >> -- Hal >> >>> >>> Thanks! >>> >>> Yicheng Jia >>> >>> >>> >>> >>> _____________________________________________________________________________ >>> Scanned by IBM Email Security Management Services powered by MessageLabs. >>> For more information please visit http://www.ers.ibm.com >>> >>> >>> _____________________________________________________________________________ >>> >>> _______________________________________________ >>> general mailing list >>> general at lists.openfabrics.org >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>> >>> To unsubscribe, please visit >>> http://openib.org/mailman/listinfo/openib-general >>> >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> >> _____________________________________________________________________________ >> >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> >> _____________________________________________________________________________ >> > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hal.rosenstock at gmail.com Thu Mar 26 10:53:50 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 26 Mar 2009 13:53:50 -0400 Subject: ***SPAM*** Re: [ofa-general] OpenSM initialization error In-Reply-To: References: Message-ID: On Thu, Mar 26, 2009 at 1:46 PM, Yicheng Jia wrote: > >> You shouldn't be running different flavor SMs on the same subnet. > The OpenSM will go to standby if there's another SM exists. Some SM will go to standby but that's not the issues with mixing SMs are well beyond the simple election/handover protocol. >> Are you sure there's no SM running here other than OpenSM ? > Actually this error always happens on Qlogic unmanaged switch, which I > believe no SM running on the subnet. Do you mean on or with QLogic switch ? What does saquery -s say ? -- Hal > It always takes more than 1 minute for the error disappear. > Thanks! > > Yicheng Jia > > > > > Hal Rosenstock > > 03/26/2009 12:13 PM > > To > Yicheng Jia > cc > general at lists.openfabrics.org, hnrose at comcast.net > Subject > Re: [ofa-general] OpenSM initialization error > > > > > On Thu, Mar 26, 2009 at 12:01 PM, Yicheng Jia wrote: >> >> LID 1 is the node that OpenSM runs on. Cisco SFS7000P is managed switch, >> SM >> is running all the time. > > You shouldn't be running different flavor SMs on the same subnet. > >> This occurs during the system starts up. At first Mthca driver 1.3 starts >> on >> all nodes, after the driver is up, the OpenSM starts on one node, then the >> error shows up. It also occurs on Qlogic unmanaged switch. > > Are you sure there's no SM running here other than OpenSM ? > > -- Hal > >> Is this related to hardware? > >> Thanks! >> >> Yicheng Jia >> >> >> >> >> Hal Rosenstock >> >> 03/26/2009 10:32 AM >> >> To >> Yicheng Jia >> cc >> general at lists.openfabrics.org, hnrose at comcast.net >> Subject >> Re: [ofa-general] OpenSM initialization error >> >> >> >> >> On Thu, Mar 26, 2009 at 11:16 AM, Yicheng Jia wrote: >>> >>> Hi Hal, >>> >>>> What is your topology ? >>> >>> Several nodes directly connect to the switch. >>> >>>> Does this message persist or go away in the log ? >>> >>> It continues reporting this message until the subnet is up, then this >>> message goes away. It lasts about 1 minute. >> >> What is LID 1 ? Is it the switch ? Is Cisco SM running ? >> >>>> Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. >>>> Amongst other changes, this message has been downgraded from error to >>>> debug in more recent versions. >>> >>> Do you know what could cause the problem? Is there any way to make the >>> error >>> time shorter? >> >> What's the exact scenario under which this occurs ? >> >> -- Hal >> >>> Thanks! >>> >>> Yicheng Jia >>> >>> >>> >>> >>> Hal Rosenstock >>> >>> 03/26/2009 09:58 AM >>> >>> To >>> Yicheng Jia >>> cc >>> hnrose at comcast.net, general at lists.openfabrics.org >>> Subject >>> Re: [ofa-general] OpenSM initialization error >>> >>> >>> >>> >>> 2009/3/25 Yicheng Jia : >>>> >>>> Hello, >>>> >>>> I run into an initialization error during OpenSM start up. The error >>>> continues for nearly 1 minutes before the subnet is up. In the log file, >>>> it >>>> reports that "__osm_ucast_mgr_process_port: ERR 3A08: No path to get to >>>> LID >>>> 1 from switch 0x66a00d90008bb". >>>> >>>> I am using Mellanox MHES18 HCA and Cisco SFS7000P switch. Do you know >>>> what >>>> the cause is and how to avoid it? >>> >>> What is your topology ? >>> >>> Does this message persist or go away in the log ? >>> >>> Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. >>> Amongst other changes, this message has been downgraded from error to >>> debug in more recent versions. >>> >>> -- Hal >>> >>>> >>>> Thanks! >>>> >>>> Yicheng Jia >>>> >>>> >>>> >>>> >>>> >>>> _____________________________________________________________________________ >>>> Scanned by IBM Email Security Management Services powered by >>>> MessageLabs. >>>> For more information please visit http://www.ers.ibm.com >>>> >>>> >>>> >>>> _____________________________________________________________________________ >>>> >>>> _______________________________________________ >>>> general mailing list >>>> general at lists.openfabrics.org >>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>>> >>>> To unsubscribe, please visit >>>> http://openib.org/mailman/listinfo/openib-general >>>> >>> >>> >>> >>> _____________________________________________________________________________ >>> Scanned by IBM Email Security Management Services powered by MessageLabs. >>> For more information please visit http://www.ers.ibm.com >>> >>> >>> _____________________________________________________________________________ >>> >>> >>> >>> >>> _____________________________________________________________________________ >>> Scanned by IBM Email Security Management Services powered by MessageLabs. >>> For more information please visit http://www.ers.ibm.com >>> >>> >>> _____________________________________________________________________________ >>> >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> >> _____________________________________________________________________________ >> >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> >> _____________________________________________________________________________ >> > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > From YJia at tmriusa.com Thu Mar 26 12:25:21 2009 From: YJia at tmriusa.com (Yicheng Jia) Date: Thu, 26 Mar 2009 14:25:21 -0500 Subject: [ofa-general] OpenSM initialization error In-Reply-To: Message-ID: It occurs with Qlogic switch, the error is reported on the OpenSM log file. Both OpenSM and mthca driver are running on QNX environment, so there's no handy tool available. What else can I do? Thanks! Yicheng Jia Hal Rosenstock 03/26/2009 12:53 PM To Yicheng Jia cc general at lists.openfabrics.org, hnrose at comcast.net Subject Re: [ofa-general] OpenSM initialization error On Thu, Mar 26, 2009 at 1:46 PM, Yicheng Jia wrote: > >> You shouldn't be running different flavor SMs on the same subnet. > The OpenSM will go to standby if there's another SM exists. Some SM will go to standby but that's not the issues with mixing SMs are well beyond the simple election/handover protocol. >> Are you sure there's no SM running here other than OpenSM ? > Actually this error always happens on Qlogic unmanaged switch, which I > believe no SM running on the subnet. Do you mean on or with QLogic switch ? What does saquery -s say ? -- Hal > It always takes more than 1 minute for the error disappear. > Thanks! > > Yicheng Jia > > > > > Hal Rosenstock > > 03/26/2009 12:13 PM > > To > Yicheng Jia > cc > general at lists.openfabrics.org, hnrose at comcast.net > Subject > Re: [ofa-general] OpenSM initialization error > > > > > On Thu, Mar 26, 2009 at 12:01 PM, Yicheng Jia wrote: >> >> LID 1 is the node that OpenSM runs on. Cisco SFS7000P is managed switch, >> SM >> is running all the time. > > You shouldn't be running different flavor SMs on the same subnet. > >> This occurs during the system starts up. At first Mthca driver 1.3 starts >> on >> all nodes, after the driver is up, the OpenSM starts on one node, then the >> error shows up. It also occurs on Qlogic unmanaged switch. > > Are you sure there's no SM running here other than OpenSM ? > > -- Hal > >> Is this related to hardware? > >> Thanks! >> >> Yicheng Jia >> >> >> >> >> Hal Rosenstock >> >> 03/26/2009 10:32 AM >> >> To >> Yicheng Jia >> cc >> general at lists.openfabrics.org, hnrose at comcast.net >> Subject >> Re: [ofa-general] OpenSM initialization error >> >> >> >> >> On Thu, Mar 26, 2009 at 11:16 AM, Yicheng Jia wrote: >>> >>> Hi Hal, >>> >>>> What is your topology ? >>> >>> Several nodes directly connect to the switch. >>> >>>> Does this message persist or go away in the log ? >>> >>> It continues reporting this message until the subnet is up, then this >>> message goes away. It lasts about 1 minute. >> >> What is LID 1 ? Is it the switch ? Is Cisco SM running ? >> >>>> Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. >>>> Amongst other changes, this message has been downgraded from error to >>>> debug in more recent versions. >>> >>> Do you know what could cause the problem? Is there any way to make the >>> error >>> time shorter? >> >> What's the exact scenario under which this occurs ? >> >> -- Hal >> >>> Thanks! >>> >>> Yicheng Jia >>> >>> >>> >>> >>> Hal Rosenstock >>> >>> 03/26/2009 09:58 AM >>> >>> To >>> Yicheng Jia >>> cc >>> hnrose at comcast.net, general at lists.openfabrics.org >>> Subject >>> Re: [ofa-general] OpenSM initialization error >>> >>> >>> >>> >>> 2009/3/25 Yicheng Jia : >>>> >>>> Hello, >>>> >>>> I run into an initialization error during OpenSM start up. The error >>>> continues for nearly 1 minutes before the subnet is up. In the log file, >>>> it >>>> reports that "__osm_ucast_mgr_process_port: ERR 3A08: No path to get to >>>> LID >>>> 1 from switch 0x66a00d90008bb". >>>> >>>> I am using Mellanox MHES18 HCA and Cisco SFS7000P switch. Do you know >>>> what >>>> the cause is and how to avoid it? >>> >>> What is your topology ? >>> >>> Does this message persist or go away in the log ? >>> >>> Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. >>> Amongst other changes, this message has been downgraded from error to >>> debug in more recent versions. >>> >>> -- Hal >>> >>>> >>>> Thanks! >>>> >>>> Yicheng Jia >>>> >>>> >>>> >>>> >>>> >>>> _____________________________________________________________________________ >>>> Scanned by IBM Email Security Management Services powered by >>>> MessageLabs. >>>> For more information please visit http://www.ers.ibm.com >>>> >>>> >>>> >>>> _____________________________________________________________________________ >>>> >>>> _______________________________________________ >>>> general mailing list >>>> general at lists.openfabrics.org >>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>>> >>>> To unsubscribe, please visit >>>> http://openib.org/mailman/listinfo/openib-general >>>> >>> >>> >>> >>> _____________________________________________________________________________ >>> Scanned by IBM Email Security Management Services powered by MessageLabs. >>> For more information please visit http://www.ers.ibm.com >>> >>> >>> _____________________________________________________________________________ >>> >>> >>> >>> >>> _____________________________________________________________________________ >>> Scanned by IBM Email Security Management Services powered by MessageLabs. >>> For more information please visit http://www.ers.ibm.com >>> >>> >>> _____________________________________________________________________________ >>> >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> >> _____________________________________________________________________________ >> >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> >> _____________________________________________________________________________ >> > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hal.rosenstock at gmail.com Thu Mar 26 13:06:10 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Thu, 26 Mar 2009 16:06:10 -0400 Subject: ***SPAM*** Re: [ofa-general] OpenSM initialization error In-Reply-To: References: Message-ID: On Thu, Mar 26, 2009 at 3:25 PM, Yicheng Jia wrote: > > It occurs with Qlogic switch, the error is reported on the OpenSM log file. > Both OpenSM and mthca driver are running on QNX environment, so there's no > handy tool available. What else can I do? Do you have a Linux end node ? If so, saquery -s there will tell how many SMs are on your subnet. -- Hal > Thanks! > > Yicheng Jia > > > > > Hal Rosenstock > > 03/26/2009 12:53 PM > > To > Yicheng Jia > cc > general at lists.openfabrics.org, hnrose at comcast.net > Subject > Re: [ofa-general] OpenSM initialization error > > > > > On Thu, Mar 26, 2009 at 1:46 PM, Yicheng Jia wrote: >> >>> You shouldn't be running different flavor SMs on the same subnet. >> The OpenSM will go to standby if there's another SM exists. > > Some SM will go to standby but that's not the issues with mixing SMs > are well beyond the simple election/handover protocol. > >>> Are you sure there's no SM running here other than OpenSM ? >> Actually this error always happens on Qlogic unmanaged switch, which I >> believe no SM running on the subnet. > > Do you mean on or with QLogic switch ? > > What does saquery -s say ? > > -- Hal > >> It always takes more than 1 minute for the error disappear. > >> Thanks! >> >> Yicheng Jia >> >> >> >> >> Hal Rosenstock >> >> 03/26/2009 12:13 PM >> >> To >> Yicheng Jia >> cc >> general at lists.openfabrics.org, hnrose at comcast.net >> Subject >> Re: [ofa-general] OpenSM initialization error >> >> >> >> >> On Thu, Mar 26, 2009 at 12:01 PM, Yicheng Jia wrote: >>> >>> LID 1 is the node that OpenSM runs on. Cisco SFS7000P is managed switch, >>> SM >>> is running all the time. >> >> You shouldn't be running different flavor SMs on the same subnet. >> >>> This occurs during the system starts up. At first Mthca driver 1.3 starts >>> on >>> all nodes, after the driver is up, the OpenSM starts on one node, then >>> the >>> error shows up. It also occurs on Qlogic unmanaged switch. >> >> Are you sure there's no SM running here other than OpenSM ? >> >> -- Hal >> >>> Is this related to hardware? >> >>> Thanks! >>> >>> Yicheng Jia >>> >>> >>> >>> >>> Hal Rosenstock >>> >>> 03/26/2009 10:32 AM >>> >>> To >>> Yicheng Jia >>> cc >>> general at lists.openfabrics.org, hnrose at comcast.net >>> Subject >>> Re: [ofa-general] OpenSM initialization error >>> >>> >>> >>> >>> On Thu, Mar 26, 2009 at 11:16 AM, Yicheng Jia wrote: >>>> >>>> Hi Hal, >>>> >>>>> What is your topology ? >>>> >>>> Several nodes directly connect to the switch. >>>> >>>>> Does this message persist or go away in the log ? >>>> >>>> It continues reporting this message until the subnet is up, then this >>>> message goes away. It lasts about 1 minute. >>> >>> What is LID 1 ? Is it the switch ? Is Cisco SM running ? >>> >>>>> Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. >>>>> Amongst other changes, this message has been downgraded from error to >>>>> debug in more recent versions. >>>> >>>> Do you know what could cause the problem? Is there any way to make the >>>> error >>>> time shorter? >>> >>> What's the exact scenario under which this occurs ? >>> >>> -- Hal >>> >>>> Thanks! >>>> >>>> Yicheng Jia >>>> >>>> >>>> >>>> >>>> Hal Rosenstock >>>> >>>> 03/26/2009 09:58 AM >>>> >>>> To >>>> Yicheng Jia >>>> cc >>>> hnrose at comcast.net, general at lists.openfabrics.org >>>> Subject >>>> Re: [ofa-general] OpenSM initialization error >>>> >>>> >>>> >>>> >>>> 2009/3/25 Yicheng Jia : >>>>> >>>>> Hello, >>>>> >>>>> I run into an initialization error during OpenSM start up. The error >>>>> continues for nearly 1 minutes before the subnet is up. In the log >>>>> file, >>>>> it >>>>> reports that "__osm_ucast_mgr_process_port: ERR 3A08: No path to get to >>>>> LID >>>>> 1 from switch 0x66a00d90008bb". >>>>> >>>>> I am using Mellanox MHES18 HCA and Cisco SFS7000P switch. Do you know >>>>> what >>>>> the cause is and how to avoid it? >>>> >>>> What is your topology ? >>>> >>>> Does this message persist or go away in the log ? >>>> >>>> Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. >>>> Amongst other changes, this message has been downgraded from error to >>>> debug in more recent versions. >>>> >>>> -- Hal >>>> >>>>> >>>>> Thanks! >>>>> >>>>> Yicheng Jia >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _____________________________________________________________________________ >>>>> Scanned by IBM Email Security Management Services powered by >>>>> MessageLabs. >>>>> For more information please visit http://www.ers.ibm.com >>>>> >>>>> >>>>> >>>>> >>>>> _____________________________________________________________________________ >>>>> >>>>> _______________________________________________ >>>>> general mailing list >>>>> general at lists.openfabrics.org >>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>>>> >>>>> To unsubscribe, please visit >>>>> http://openib.org/mailman/listinfo/openib-general >>>>> >>>> >>>> >>>> >>>> >>>> _____________________________________________________________________________ >>>> Scanned by IBM Email Security Management Services powered by >>>> MessageLabs. >>>> For more information please visit http://www.ers.ibm.com >>>> >>>> >>>> >>>> _____________________________________________________________________________ >>>> >>>> >>>> >>>> >>>> >>>> _____________________________________________________________________________ >>>> Scanned by IBM Email Security Management Services powered by >>>> MessageLabs. >>>> For more information please visit http://www.ers.ibm.com >>>> >>>> >>>> >>>> _____________________________________________________________________________ >>>> >>> >>> >>> >>> _____________________________________________________________________________ >>> Scanned by IBM Email Security Management Services powered by MessageLabs. >>> For more information please visit http://www.ers.ibm.com >>> >>> >>> _____________________________________________________________________________ >>> >>> >>> >>> >>> _____________________________________________________________________________ >>> Scanned by IBM Email Security Management Services powered by MessageLabs. >>> For more information please visit http://www.ers.ibm.com >>> >>> >>> _____________________________________________________________________________ >>> >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> >> _____________________________________________________________________________ >> >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> >> _____________________________________________________________________________ >> > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > From jon at opengridcomputing.com Thu Mar 26 13:19:05 2009 From: jon at opengridcomputing.com (Jon Mason) Date: Thu, 26 Mar 2009 15:19:05 -0500 Subject: [ofa-general] NFS-RDMA server run out-of-memory, In-Reply-To: <49CBB9A1.9020209@mellanox.com> References: <49CBB9A1.9020209@mellanox.com> Message-ID: <20090326201905.GB3571@opengridcomputing.com> On Thu, Mar 26, 2009 at 10:21:37AM -0700, Vu Pham wrote: > Jon, > > nfsrdma client (RHEL 5.2), Arbel or ConnectX HCAs > nfsrdma server (RHEL 5.2/5.3 or 2.6.27), Arbel HCA > > I run connectathon -N 1000 (or less if you have less memory on nfsrdma server), > System run out-of-memory and reboot. > > Do you see the same behavior with Chelsio? I see the same behavior on Chelsio over RDMA and over TCP. > Call Trace: > [] out_of_memory+0x8e/0x2f5 > [] __alloc_pages+0x245/0x2ce > [] alloc_page_interleave+0x3d/0x74 > [] __do_page_cache_readahead+0x95/0x1d9 > [] zone_statistics+0x3e/0x6d > [] ifind_fast+0x47/0x83 > [] iput+0x4b/0x84 > [] blockable_page_cache_readahead+0x53/0xb2 > [] page_cache_readahead+0x13d/0x1af > [] do_generic_mapping_read+0x126/0x3f8 > [] :nfsd:nfsd_read_actor+0x0/0xd9 > [] generic_file_sendfile+0x4c/0x64 > [] :nfsd:nfsd_vfs_read+0x20a/0x32f > [] :nfsd:nfsd_read+0x9c/0xba > [] :nfsd:nfsd3_proc_read+0x11b/0x161 > [] :nfsd:nfsd_dispatch+0xde/0x1b6 > [] :sunrpc:svc_process+0x405/0x6de > [] __down_read+0x12/0x92 > [] keventd_create_kthread+0x0/0xc4 > [] :nfsd:nfsd+0x1a3/0x274 > [] :nfsd:nfsd+0x0/0x274 > [] kthread+0xfe/0x132 > [] child_rip+0xa/0x11 > [] keventd_create_kthread+0x0/0xc4 > [] kthread+0x0/0x132 > [] child_rip+0x0/0x11 > > Node 0 DMA per-cpu: > cpu 0 hot: high 0, batch 1 used:0 > cpu 0 cold: high 0, batch 1 used:0 > cpu 1 hot: high 0, batch 1 used:0 > cpu 1 cold: high 0, batch 1 used:0 > cpu 2 hot: high 0, batch 1 used:0 > cpu 2 cold: high 0, batch 1 used:0 > cpu 3 hot: high 0, batch 1 used:0 > cpu 3 cold: high 0, batch 1 used:0 > Node 0 DMA32 per-cpu: > > > Call Trace: > [] out_of_memory+0x8e/0x2f5 > [] __alloc_pages+0x245/0x2ce > [] _spin_lock_bh+0x9/0x14 > [] alloc_page_interleave+0x3d/0x74 > [] :sunrpc:svc_recv+0xc2/0x73f > [] default_wake_function+0x0/0xe > [] __down_read+0x12/0x92 > [] keventd_create_kthread+0x0/0xc4 > [] :nfsd:nfsd+0xd3/0x274 > [] :nfsd:nfsd+0x0/0x274 > [] kthread+0xfe/0x132 > [] child_rip+0xa/0x11 > [] keventd_create_kthread+0x0/0xc4 > [] kthread+0x0/0x132 > [] child_rip+0x0/0x11 > > > -vu > From vlad at lists.openfabrics.org Fri Mar 27 03:26:27 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Fri, 27 Mar 2009 03:26:27 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090327-0200 daily build status Message-ID: <20090327102627.99644E61153@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From YJia at tmriusa.com Fri Mar 27 12:55:46 2009 From: YJia at tmriusa.com (Yicheng Jia) Date: Fri, 27 Mar 2009 14:55:46 -0500 Subject: [ofa-general] OpenSM initialization error In-Reply-To: Message-ID: Here is the output of saquery. I am using Qlogic 9024 FC unmanaged switch. Only one SM exists in the subnet. # saquery -s IsSM ports PortInfoRecord dump: EndPortLid..............0x1 PortNum.................0x1 base_lid................0x1 master_sm_base_lid......0x1 capability_mask.........0x2500A6A IsSMdisabled ports Thanks! Yicheng Jia Hal Rosenstock 03/26/2009 03:06 PM To Yicheng Jia cc general at lists.openfabrics.org, hnrose at comcast.net Subject Re: [ofa-general] OpenSM initialization error On Thu, Mar 26, 2009 at 3:25 PM, Yicheng Jia wrote: > > It occurs with Qlogic switch, the error is reported on the OpenSM log file. > Both OpenSM and mthca driver are running on QNX environment, so there's no > handy tool available. What else can I do? Do you have a Linux end node ? If so, saquery -s there will tell how many SMs are on your subnet. -- Hal > Thanks! > > Yicheng Jia > > > > > Hal Rosenstock > > 03/26/2009 12:53 PM > > To > Yicheng Jia > cc > general at lists.openfabrics.org, hnrose at comcast.net > Subject > Re: [ofa-general] OpenSM initialization error > > > > > On Thu, Mar 26, 2009 at 1:46 PM, Yicheng Jia wrote: >> >>> You shouldn't be running different flavor SMs on the same subnet. >> The OpenSM will go to standby if there's another SM exists. > > Some SM will go to standby but that's not the issues with mixing SMs > are well beyond the simple election/handover protocol. > >>> Are you sure there's no SM running here other than OpenSM ? >> Actually this error always happens on Qlogic unmanaged switch, which I >> believe no SM running on the subnet. > > Do you mean on or with QLogic switch ? > > What does saquery -s say ? > > -- Hal > >> It always takes more than 1 minute for the error disappear. > >> Thanks! >> >> Yicheng Jia >> >> >> >> >> Hal Rosenstock >> >> 03/26/2009 12:13 PM >> >> To >> Yicheng Jia >> cc >> general at lists.openfabrics.org, hnrose at comcast.net >> Subject >> Re: [ofa-general] OpenSM initialization error >> >> >> >> >> On Thu, Mar 26, 2009 at 12:01 PM, Yicheng Jia wrote: >>> >>> LID 1 is the node that OpenSM runs on. Cisco SFS7000P is managed switch, >>> SM >>> is running all the time. >> >> You shouldn't be running different flavor SMs on the same subnet. >> >>> This occurs during the system starts up. At first Mthca driver 1.3 starts >>> on >>> all nodes, after the driver is up, the OpenSM starts on one node, then >>> the >>> error shows up. It also occurs on Qlogic unmanaged switch. >> >> Are you sure there's no SM running here other than OpenSM ? >> >> -- Hal >> >>> Is this related to hardware? >> >>> Thanks! >>> >>> Yicheng Jia >>> >>> >>> >>> >>> Hal Rosenstock >>> >>> 03/26/2009 10:32 AM >>> >>> To >>> Yicheng Jia >>> cc >>> general at lists.openfabrics.org, hnrose at comcast.net >>> Subject >>> Re: [ofa-general] OpenSM initialization error >>> >>> >>> >>> >>> On Thu, Mar 26, 2009 at 11:16 AM, Yicheng Jia wrote: >>>> >>>> Hi Hal, >>>> >>>>> What is your topology ? >>>> >>>> Several nodes directly connect to the switch. >>>> >>>>> Does this message persist or go away in the log ? >>>> >>>> It continues reporting this message until the subnet is up, then this >>>> message goes away. It lasts about 1 minute. >>> >>> What is LID 1 ? Is it the switch ? Is Cisco SM running ? >>> >>>>> Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. >>>>> Amongst other changes, this message has been downgraded from error to >>>>> debug in more recent versions. >>>> >>>> Do you know what could cause the problem? Is there any way to make the >>>> error >>>> time shorter? >>> >>> What's the exact scenario under which this occurs ? >>> >>> -- Hal >>> >>>> Thanks! >>>> >>>> Yicheng Jia >>>> >>>> >>>> >>>> >>>> Hal Rosenstock >>>> >>>> 03/26/2009 09:58 AM >>>> >>>> To >>>> Yicheng Jia >>>> cc >>>> hnrose at comcast.net, general at lists.openfabrics.org >>>> Subject >>>> Re: [ofa-general] OpenSM initialization error >>>> >>>> >>>> >>>> >>>> 2009/3/25 Yicheng Jia : >>>>> >>>>> Hello, >>>>> >>>>> I run into an initialization error during OpenSM start up. The error >>>>> continues for nearly 1 minutes before the subnet is up. In the log >>>>> file, >>>>> it >>>>> reports that "__osm_ucast_mgr_process_port: ERR 3A08: No path to get to >>>>> LID >>>>> 1 from switch 0x66a00d90008bb". >>>>> >>>>> I am using Mellanox MHES18 HCA and Cisco SFS7000P switch. Do you know >>>>> what >>>>> the cause is and how to avoid it? >>>> >>>> What is your topology ? >>>> >>>> Does this message persist or go away in the log ? >>>> >>>> Looks to me like you are using an OpenSM 3.1.11/OFED 1.3 or older. >>>> Amongst other changes, this message has been downgraded from error to >>>> debug in more recent versions. >>>> >>>> -- Hal >>>> >>>>> >>>>> Thanks! >>>>> >>>>> Yicheng Jia >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _____________________________________________________________________________ >>>>> Scanned by IBM Email Security Management Services powered by >>>>> MessageLabs. >>>>> For more information please visit http://www.ers.ibm.com >>>>> >>>>> >>>>> >>>>> >>>>> _____________________________________________________________________________ >>>>> >>>>> _______________________________________________ >>>>> general mailing list >>>>> general at lists.openfabrics.org >>>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >>>>> >>>>> To unsubscribe, please visit >>>>> http://openib.org/mailman/listinfo/openib-general >>>>> >>>> >>>> >>>> >>>> >>>> _____________________________________________________________________________ >>>> Scanned by IBM Email Security Management Services powered by >>>> MessageLabs. >>>> For more information please visit http://www.ers.ibm.com >>>> >>>> >>>> >>>> _____________________________________________________________________________ >>>> >>>> >>>> >>>> >>>> >>>> _____________________________________________________________________________ >>>> Scanned by IBM Email Security Management Services powered by >>>> MessageLabs. >>>> For more information please visit http://www.ers.ibm.com >>>> >>>> >>>> >>>> _____________________________________________________________________________ >>>> >>> >>> >>> >>> _____________________________________________________________________________ >>> Scanned by IBM Email Security Management Services powered by MessageLabs. >>> For more information please visit http://www.ers.ibm.com >>> >>> >>> _____________________________________________________________________________ >>> >>> >>> >>> >>> _____________________________________________________________________________ >>> Scanned by IBM Email Security Management Services powered by MessageLabs. >>> For more information please visit http://www.ers.ibm.com >>> >>> >>> _____________________________________________________________________________ >>> >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> >> _____________________________________________________________________________ >> >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> >> _____________________________________________________________________________ >> > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hal.rosenstock at gmail.com Fri Mar 27 13:02:23 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 27 Mar 2009 16:02:23 -0400 Subject: ***SPAM*** Re: [ofa-general] OpenSM initialization error In-Reply-To: References: Message-ID: On Fri, Mar 27, 2009 at 3:55 PM, Yicheng Jia wrote: > > Here is the output of saquery. I am using Qlogic 9024 FC unmanaged switch. > Only one SM exists in the subnet. Is this topology simply one switch surrounded by HCAs ? If so, I don't have a theory as to why this message persists for 1 second. -- Hal From YJia at tmriusa.com Fri Mar 27 13:13:39 2009 From: YJia at tmriusa.com (Yicheng Jia) Date: Fri, 27 Mar 2009 15:13:39 -0500 Subject: [ofa-general] OpenSM initialization error In-Reply-To: Message-ID: Yes. This persists for nearly 1 minute. One thing I notice is if I start mthca driver more than 1 minute before OpenSM starts, then there's no such error message anymore, this is gone. So I guess there's some handshaking going on between mthca driver and unmanaged switch firmware, which stablize communication before OpenSM starts. Do you have any idea about it? Thanks! Yicheng Jia Hal Rosenstock 03/27/2009 03:02 PM To Yicheng Jia cc general at lists.openfabrics.org, hnrose at comcast.net Subject Re: [ofa-general] OpenSM initialization error On Fri, Mar 27, 2009 at 3:55 PM, Yicheng Jia wrote: > > Here is the output of saquery. I am using Qlogic 9024 FC unmanaged switch. > Only one SM exists in the subnet. Is this topology simply one switch surrounded by HCAs ? If so, I don't have a theory as to why this message persists for 1 second. -- Hal _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From hal.rosenstock at gmail.com Fri Mar 27 13:18:00 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 27 Mar 2009 16:18:00 -0400 Subject: ***SPAM*** Re: [ofa-general] OpenSM initialization error In-Reply-To: References: Message-ID: On Fri, Mar 27, 2009 at 4:13 PM, Yicheng Jia wrote: > > Yes. This persists for nearly 1 minute. > > One thing I notice is if I start mthca driver more than 1 minute before > OpenSM starts, then there's no such error message anymore, this is gone. So > I guess there's some handshaking going on between mthca driver and unmanaged > switch firmware, which stablize communication before OpenSM starts. Do you > have any idea about it? The physical link is brought up all on its own assuming the ports are not disabled. Maybe the SM link being used is not in init yet. I don't think that is well handled. -- Hal > Thanks! > > Yicheng Jia > > > > > Hal Rosenstock > > 03/27/2009 03:02 PM > > To > Yicheng Jia > cc > general at lists.openfabrics.org, hnrose at comcast.net > Subject > Re: [ofa-general] OpenSM initialization error > > > > > On Fri, Mar 27, 2009 at 3:55 PM, Yicheng Jia wrote: >> >> Here is the output of saquery. I am using Qlogic 9024 FC unmanaged switch. >> Only one SM exists in the subnet. > > > > Is this topology simply one switch surrounded by HCAs ? If so, I don't > have a theory as to why this message persists for 1 second. > > -- Hal > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit > http://www.ers.ibm.com > _____________________________________________________________________________ > > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > From hal.rosenstock at gmail.com Fri Mar 27 13:19:21 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 27 Mar 2009 16:19:21 -0400 Subject: ***SPAM*** Re: [ofa-general] OpenSM initialization error In-Reply-To: References: Message-ID: On Fri, Mar 27, 2009 at 4:18 PM, Hal Rosenstock wrote: > On Fri, Mar 27, 2009 at 4:13 PM, Yicheng Jia wrote: >> >> Yes. This persists for nearly 1 minute. >> >> One thing I notice is if I start mthca driver more than 1 minute before >> OpenSM starts, then there's no such error message anymore, this is gone. So >> I guess there's some handshaking going on between mthca driver and unmanaged >> switch firmware, which stablize communication before OpenSM starts. Do you >> have any idea about it? > > The physical link is brought up all on its own assuming the ports are > not disabled. Maybe the SM link being used is not in init yet. I don't > think that is well handled. What do the port counters say on both sides of the SM link ? Maybe try different cables on the SM link and see if this gets better. -- Hal > -- Hal > >> Thanks! >> >> Yicheng Jia >> >> >> >> >> Hal Rosenstock >> >> 03/27/2009 03:02 PM >> >> To >> Yicheng Jia >> cc >> general at lists.openfabrics.org, hnrose at comcast.net >> Subject >> Re: [ofa-general] OpenSM initialization error >> >> >> >> >> On Fri, Mar 27, 2009 at 3:55 PM, Yicheng Jia wrote: >>> >>> Here is the output of saquery. I am using Qlogic 9024 FC unmanaged switch. >>> Only one SM exists in the subnet. >> >> >> >> Is this topology simply one switch surrounded by HCAs ? If so, I don't >> have a theory as to why this message persists for 1 second. >> >> -- Hal >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit >> http://www.ers.ibm.com >> _____________________________________________________________________________ >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> _____________________________________________________________________________ >> > From YJia at tmriusa.com Fri Mar 27 13:51:42 2009 From: YJia at tmriusa.com (Yicheng Jia) Date: Fri, 27 Mar 2009 15:51:42 -0500 Subject: [ofa-general] OpenSM initialization error In-Reply-To: Message-ID: The "port_rcv_errors" attribute on OpenSM side of SM link is 65535, other errors attributes are all 0. How can I check SM link status on the switch side? Is there anything I can do if the SM link port on the switch side is not in init state? I don't think it's cable issue. I have 3 unmanaged switches on different system, all have the same problem. Thanks! Yicheng Jia Hal Rosenstock 03/27/2009 03:19 PM To Yicheng Jia cc general at lists.openfabrics.org, hnrose at comcast.net Subject Re: [ofa-general] OpenSM initialization error On Fri, Mar 27, 2009 at 4:18 PM, Hal Rosenstock wrote: > On Fri, Mar 27, 2009 at 4:13 PM, Yicheng Jia wrote: >> >> Yes. This persists for nearly 1 minute. >> >> One thing I notice is if I start mthca driver more than 1 minute before >> OpenSM starts, then there's no such error message anymore, this is gone. So >> I guess there's some handshaking going on between mthca driver and unmanaged >> switch firmware, which stablize communication before OpenSM starts. Do you >> have any idea about it? > > The physical link is brought up all on its own assuming the ports are > not disabled. Maybe the SM link being used is not in init yet. I don't > think that is well handled. What do the port counters say on both sides of the SM link ? Maybe try different cables on the SM link and see if this gets better. -- Hal > -- Hal > >> Thanks! >> >> Yicheng Jia >> >> >> >> >> Hal Rosenstock >> >> 03/27/2009 03:02 PM >> >> To >> Yicheng Jia >> cc >> general at lists.openfabrics.org, hnrose at comcast.net >> Subject >> Re: [ofa-general] OpenSM initialization error >> >> >> >> >> On Fri, Mar 27, 2009 at 3:55 PM, Yicheng Jia wrote: >>> >>> Here is the output of saquery. I am using Qlogic 9024 FC unmanaged switch. >>> Only one SM exists in the subnet. >> >> >> >> Is this topology simply one switch surrounded by HCAs ? If so, I don't >> have a theory as to why this message persists for 1 second. >> >> -- Hal >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit >> http://www.ers.ibm.com >> _____________________________________________________________________________ >> >> >> _____________________________________________________________________________ >> Scanned by IBM Email Security Management Services powered by MessageLabs. >> For more information please visit http://www.ers.ibm.com >> _____________________________________________________________________________ >> > _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.hefty at intel.com Fri Mar 27 14:36:56 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Fri, 27 Mar 2009 14:36:56 -0700 Subject: [ofa-general] RE: [PATCH] rdma_cm: create cm id even when port is down In-Reply-To: <49CA88A7.7060006@Voltaire.COM> References: <49CA88A7.7060006@Voltaire.COM> Message-ID: > When doing rdma_resolve_addr() and relevant port is down, the function fails >and rdma_cm id is not bound to the device. Therefore, application does not have >device handle and cannot wait for the port to become active. The function >fails because ipoib is not joined to the multicast group and therefore sa does >not have a multicast record to take a qkey from. > The patch here is to make lazy qkey resolution - cma_set_qkey will set >id_priv->qkey if it was not set, and will be called just before the qkey is >really required. > >Signed-off-by: Yossi Etigin Acked-by: Sean Hefty >--- Roland, a thread that discussed this starts here: http://lists.openfabrics.org/pipermail/general/2009-February/056895.html The subject never contained '[PATCH]', so it was probably missed, but Yossi's patch should be good for 2.6.30. > drivers/infiniband/core/cma.c | 41 +++++++++++++++++++++++++++-------------- > 1 file changed, 27 insertions(+), 14 deletions(-) > >Index: b/drivers/infiniband/core/cma.c >=================================================================== >--- a/drivers/infiniband/core/cma.c 2009-03-10 18:21:47.000000000 +0200 >+++ b/drivers/infiniband/core/cma.c 2009-03-10 19:22:18.000000000 +0200 >@@ -297,21 +297,25 @@ static void cma_detach_from_dev(struct r > id_priv->cma_dev = NULL; > } > >-static int cma_set_qkey(struct ib_device *device, u8 port_num, >- enum rdma_port_space ps, >- struct rdma_dev_addr *dev_addr, u32 *qkey) >+static int cma_set_qkey(struct rdma_id_private *id_priv) > { > struct ib_sa_mcmember_rec rec; > int ret = 0; > >- switch (ps) { >+ if (id_priv->qkey) >+ return; >+ >+ switch (id_priv->id.ps) { > case RDMA_PS_UDP: >- *qkey = RDMA_UDP_QKEY; >+ id_priv->qkey = RDMA_UDP_QKEY; > break; > case RDMA_PS_IPOIB: >- ib_addr_get_mgid(dev_addr, &rec.mgid); >- ret = ib_sa_get_mcmember_rec(device, port_num, &rec.mgid, &rec); >- *qkey = be32_to_cpu(rec.qkey); >+ ib_addr_get_mgid(&id_priv->id.route.addr.dev_addr, &rec.mgid); >+ ret = ib_sa_get_mcmember_rec(id_priv->id.device, >+ id_priv->id.port_num, &rec.mgid, >+ &rec); >+ if (!ret) >+ id_priv->qkey = be32_to_cpu(rec.qkey); > break; > default: > break; >@@ -341,12 +345,7 @@ static int cma_acquire_dev(struct rdma_i > ret = ib_find_cached_gid(cma_dev->device, &gid, > &id_priv->id.port_num, NULL); > if (!ret) { >- ret = cma_set_qkey(cma_dev->device, >- id_priv->id.port_num, >- id_priv->id.ps, dev_addr, >- &id_priv->qkey); >- if (!ret) >- cma_attach_to_dev(id_priv, cma_dev); >+ cma_attach_to_dev(id_priv, cma_dev); > break; > } > } >@@ -578,6 +577,10 @@ static int cma_ib_init_qp_attr(struct rd > *qp_attr_mask = IB_QP_STATE | IB_QP_PKEY_INDEX | IB_QP_PORT; > > if (cma_is_ud_ps(id_priv->id.ps)) { >+ ret = cma_set_qkey(id_priv); >+ if (ret) >+ return ret; >+ > qp_attr->qkey = id_priv->qkey; > *qp_attr_mask |= IB_QP_QKEY; > } else { >@@ -2201,6 +2204,12 @@ static int cma_sidr_rep_handler(struct i > event.status = ib_event->param.sidr_rep_rcvd.status; > break; > } >+ ret = cma_set_qkey(id_priv); >+ if (ret) { >+ event.event = RDMA_CM_EVENT_ADDR_ERROR; >+ event.status = -EINVAL; >+ break; >+ } > if (id_priv->qkey != rep->qkey) { > event.event = RDMA_CM_EVENT_UNREACHABLE; > event.status = -EINVAL; >@@ -2480,10 +2489,14 @@ static int cma_send_sidr_rep(struct rdma > const void *private_data, int private_data_len) > { > struct ib_cm_sidr_rep_param rep; >+ int ret; > > memset(&rep, 0, sizeof rep); > rep.status = status; > if (status == IB_SIDR_SUCCESS) { >+ ret = cma_set_qkey(id_priv); >+ if (ret) >+ return ret; > rep.qp_num = id_priv->qp_num; > rep.qkey = id_priv->qkey; > } From hal.rosenstock at gmail.com Fri Mar 27 14:55:39 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Fri, 27 Mar 2009 17:55:39 -0400 Subject: ***SPAM*** Re: [ofa-general] OpenSM initialization error In-Reply-To: References: Message-ID: On Fri, Mar 27, 2009 at 4:51 PM, Yicheng Jia wrote: > > The "port_rcv_errors" attribute on OpenSM side of SM link is 65535, other > errors attributes are all 0. That's max'd out so should be cleared to see if it is continually incrementing. Port Rcv Errors are the following: • Local physical errors (ICRC, VCRC, LPCRC, and all physical errors that cause entry into the BAD PACKET or BAD PACKET DISCARD states of the packet receiver state machine) • Malformed data packet errors (LVer, length, VL) • Malformed link packet errors (operand, length, VL) • Packets discarded due to buffer overrun > How can I check SM link status on the switch side? Get the portinfo on the SM peer port. Something like: smpquery -D portinfo 0,1 assuming you are doing this from the SM node. > Is there anything I can do if the SM link port on the switch side is > not in init state? Let's see if that's the case first. > I don't think it's cable issue. I have 3 unmanaged switches on different > system, all have the same problem. So different cables are in use ? Same HCA for OpenSM though ? -- Hal > Thanks! > > Yicheng Jia > > > > > Hal Rosenstock > > 03/27/2009 03:19 PM > > To > Yicheng Jia > cc > general at lists.openfabrics.org, hnrose at comcast.net > Subject > Re: [ofa-general] OpenSM initialization error > > > > > On Fri, Mar 27, 2009 at 4:18 PM, Hal Rosenstock > wrote: >> On Fri, Mar 27, 2009 at 4:13 PM, Yicheng Jia wrote: >>> >>> Yes. This persists for nearly 1 minute. >>> >>> One thing I notice is if I start mthca driver more than 1 minute before >>> OpenSM starts, then there's no such error message anymore, this is gone. >>> So >>> I guess there's some handshaking going on between mthca driver and >>> unmanaged >>> switch firmware, which stablize communication before OpenSM starts. Do >>> you >>> have any idea about it? >> >> The physical link is brought up all on its own assuming the ports are >> not disabled. Maybe the SM link being used is not in init yet. I don't >> think that is well handled. > > What do the port counters say on both sides of the SM link ? > > Maybe try different cables on the SM link and see if this gets better. > > -- Hal > >> -- Hal >> >>> Thanks! >>> >>> Yicheng Jia >>> >>> >>> >>> >>> Hal Rosenstock >>> >>> 03/27/2009 03:02 PM >>> >>> To >>> Yicheng Jia >>> cc >>> general at lists.openfabrics.org, hnrose at comcast.net >>> Subject >>> Re: [ofa-general] OpenSM initialization error >>> >>> >>> >>> >>> On Fri, Mar 27, 2009 at 3:55 PM, Yicheng Jia wrote: >>>> >>>> Here is the output of saquery. I am using Qlogic 9024 FC unmanaged >>>> switch. >>>> Only one SM exists in the subnet. >>> >>> >>> >>> Is this topology simply one switch surrounded by HCAs ? If so, I don't >>> have a theory as to why this message persists for 1 second. >>> >>> -- Hal >>> >>> >>> _____________________________________________________________________________ >>> Scanned by IBM Email Security Management Services powered by MessageLabs. >>> For more information please visit >>> http://www.ers.ibm.com >>> >>> _____________________________________________________________________________ >>> >>> >>> >>> _____________________________________________________________________________ >>> Scanned by IBM Email Security Management Services powered by MessageLabs. >>> For more information please visit http://www.ers.ibm.com >>> >>> _____________________________________________________________________________ >>> >> > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > From rdreier at cisco.com Fri Mar 27 15:15:07 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 27 Mar 2009 15:15:07 -0700 Subject: [ofa-general] [PATCH] IB/mlx4: Use pgprot_writecombine() for BlueFlame pages Message-ID: The PAT work on x86 has finally made pgprot_writecombine() a usable API for modular drivers. As the comment indicates, this is exactly what we want to use in mlx4_ib to map BlueFlame pages up to userspace, since using WC for these pages improves small message latency significantly. Signed-off-by: Roland Dreier --- Could someone test this patch on top of 2.6.29, and see if it provides the right latency benefit? If it looks good, I'll send it upstream for 2.6.30. Thanks, Roland diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c index 2ccb9d3..ae3d759 100644 --- a/drivers/infiniband/hw/mlx4/main.c +++ b/drivers/infiniband/hw/mlx4/main.c @@ -394,8 +394,7 @@ static int mlx4_ib_mmap(struct ib_ucontext *context, struct vm_area_struct *vma) PAGE_SIZE, vma->vm_page_prot)) return -EAGAIN; } else if (vma->vm_pgoff == 1 && dev->dev->caps.bf_reg_size != 0) { - /* FIXME want pgprot_writecombine() for BlueFlame pages */ - vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); + vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot); if (io_remap_pfn_range(vma, vma->vm_start, to_mucontext(context)->uar.pfn + From YJia at tmriusa.com Fri Mar 27 15:20:08 2009 From: YJia at tmriusa.com (Yicheng Jia) Date: Fri, 27 Mar 2009 17:20:08 -0500 Subject: [ofa-general] OpenSM initialization error In-Reply-To: Message-ID: > That's max'd out so should be cleared to see if it is continually incrementing. The number doesn't change after the subnet is up. >Get the portinfo on the SM peer port. Something like: >smpquery -D portinfo 0,1 >assuming you are doing this from the SM node. What if I am running it on a node other than SM node? I don't have tools to run on SM node now. > So different cables are in use ? Same HCA for OpenSM though ? The cables are all the same type. HCAs are all same as well. Thanks! Yicheng Jia Hal Rosenstock 03/27/2009 04:55 PM To Yicheng Jia cc general at lists.openfabrics.org, hnrose at comcast.net Subject Re: [ofa-general] OpenSM initialization error On Fri, Mar 27, 2009 at 4:51 PM, Yicheng Jia wrote: > > The "port_rcv_errors" attribute on OpenSM side of SM link is 65535, other > errors attributes are all 0. That's max'd out so should be cleared to see if it is continually incrementing. Port Rcv Errors are the following: • Local physical errors (ICRC, VCRC, LPCRC, and all physical errors that cause entry into the BAD PACKET or BAD PACKET DISCARD states of the packet receiver state machine) • Malformed data packet errors (LVer, length, VL) • Malformed link packet errors (operand, length, VL) • Packets discarded due to buffer overrun > How can I check SM link status on the switch side? Get the portinfo on the SM peer port. Something like: smpquery -D portinfo 0,1 assuming you are doing this from the SM node. > Is there anything I can do if the SM link port on the switch side is > not in init state? Let's see if that's the case first. > I don't think it's cable issue. I have 3 unmanaged switches on different > system, all have the same problem. So different cables are in use ? Same HCA for OpenSM though ? -- Hal > Thanks! > > Yicheng Jia > > > > > Hal Rosenstock > > 03/27/2009 03:19 PM > > To > Yicheng Jia > cc > general at lists.openfabrics.org, hnrose at comcast.net > Subject > Re: [ofa-general] OpenSM initialization error > > > > > On Fri, Mar 27, 2009 at 4:18 PM, Hal Rosenstock > wrote: >> On Fri, Mar 27, 2009 at 4:13 PM, Yicheng Jia wrote: >>> >>> Yes. This persists for nearly 1 minute. >>> >>> One thing I notice is if I start mthca driver more than 1 minute before >>> OpenSM starts, then there's no such error message anymore, this is gone. >>> So >>> I guess there's some handshaking going on between mthca driver and >>> unmanaged >>> switch firmware, which stablize communication before OpenSM starts. Do >>> you >>> have any idea about it? >> >> The physical link is brought up all on its own assuming the ports are >> not disabled. Maybe the SM link being used is not in init yet. I don't >> think that is well handled. > > What do the port counters say on both sides of the SM link ? > > Maybe try different cables on the SM link and see if this gets better. > > -- Hal > >> -- Hal >> >>> Thanks! >>> >>> Yicheng Jia >>> >>> >>> >>> >>> Hal Rosenstock >>> >>> 03/27/2009 03:02 PM >>> >>> To >>> Yicheng Jia >>> cc >>> general at lists.openfabrics.org, hnrose at comcast.net >>> Subject >>> Re: [ofa-general] OpenSM initialization error >>> >>> >>> >>> >>> On Fri, Mar 27, 2009 at 3:55 PM, Yicheng Jia wrote: >>>> >>>> Here is the output of saquery. I am using Qlogic 9024 FC unmanaged >>>> switch. >>>> Only one SM exists in the subnet. >>> >>> >>> >>> Is this topology simply one switch surrounded by HCAs ? If so, I don't >>> have a theory as to why this message persists for 1 second. >>> >>> -- Hal >>> >>> >>> _____________________________________________________________________________ >>> Scanned by IBM Email Security Management Services powered by MessageLabs. >>> For more information please visit >>> http://www.ers.ibm.com >>> >>> _____________________________________________________________________________ >>> >>> >>> >>> _____________________________________________________________________________ >>> Scanned by IBM Email Security Management Services powered by MessageLabs. >>> For more information please visit http://www.ers.ibm.com >>> >>> _____________________________________________________________________________ >>> >> > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > > > _____________________________________________________________________________ > Scanned by IBM Email Security Management Services powered by MessageLabs. > For more information please visit http://www.ers.ibm.com > _____________________________________________________________________________ > _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ _____________________________________________________________________________ Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com _____________________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From phillipwils at gmail.com Fri Mar 27 16:36:16 2009 From: phillipwils at gmail.com (Phillip Wilson) Date: Fri, 27 Mar 2009 16:36:16 -0700 Subject: [ofa-general] ***SPAM*** Re: System crashed while booting Linux (ia64) with three Mellanox HCAs (15b3:6274) In-Reply-To: References: <6e4f44220903251939i68cb4268w1284bbb0552643e7@mail.gmail.com> Message-ID: <6e4f44220903271636x7095767fo92eccd72b8bca42c@mail.gmail.com> I spent the last couple of days retracing my steps. In my haste, I listed the wrong HCA firmware revision. It was firmware 1.2.940 that caused the system to crash while booting to Linux. I have the mthca driver built into the kernel; it is not a loadable driver. The system boots fine with the 1.2.0 firmware. You are correct; the system crash does not make sense since the stack was okay a few instruction earlier. I am currently looking at the error dump of the system log to see if I can find out more. There is no timing issue in the driver function ib_mad_post_receive_mads() and the debug printk messages in this function do not solve the system crash. On Thu, Mar 26, 2009 at 8:54 AM, Roland Dreier wrote: >  > System crashes with three Mellanox mezzanine cards (VID=15b3, >  > DID=0x6274) installed when booting Linux (ia64).  I am using Linux >  > 2.6.24, but this issue also occurs with Linux kernel 2.6.29-rc8. > > this is a pretty interesting crash.  Do you have the ib_mthca driver > built into your kernel, or is it being loaded as a module? > >  > A partial listing from ib_mad_post_receive_mad.S is posted below the "C" code. >  > The exact instruction that cause the system crash was located at >  > >  > ib_mad_post_*+0x0080           st4              [r2]=r3                      MII >  >                                nop.i            0x0 >  >                                nop.i            0x0 >  > >  > It tries to store r3=0x1600 to [r2] @ 0xE0000007E01C7CCC. > > Looking at the assembly, it seems the relevant parts are: > > ib_mad_post_*+0x0060           ld4              r3=[r11]                     MMI >                               st8              [r2]=r8 >                               adds             r2=28,r12 > ib_mad_post_*+0x0070           st4              [r9]=r10                     MMI >                               st8              [r45]=r0 >                               nop.i            0x0;; > ib_mad_post_*+0x0080           st4              [r2]=r3                      MII > > The main points are "adds r2=28,r12" -- ie r2 now points into the > stack -- and "st4 [r2]=r3" -- ie a store onto the stack is crashing. > > In the same function, we have "adds r9=56,r12" and "st4 [r9]=r10" > slightly earlier, so the stack isn't totally messed up (apparently). > > Not sure how to debug this since the crash as it stands doesn't seem to > make much sense... > From rdreier at cisco.com Fri Mar 27 16:48:25 2009 From: rdreier at cisco.com (Roland Dreier) Date: Fri, 27 Mar 2009 16:48:25 -0700 Subject: [ofa-general] Re: System crashed while booting Linux (ia64) with three Mellanox HCAs (15b3:6274) In-Reply-To: <6e4f44220903271636x7095767fo92eccd72b8bca42c@mail.gmail.com> (Phillip Wilson's message of "Fri, 27 Mar 2009 16:36:16 -0700") References: <6e4f44220903251939i68cb4268w1284bbb0552643e7@mail.gmail.com> <6e4f44220903271636x7095767fo92eccd72b8bca42c@mail.gmail.com> Message-ID: > I spent the last couple of days retracing my steps. In my haste, I > listed the wrong HCA firmware revision. It was firmware 1.2.940 that > caused the system to crash while booting to Linux. I have the mthca > driver built into the kernel; it is not a loadable driver. The system > boots fine with the 1.2.0 firmware. Oh, it's mthca firmware version dependent? That's a big clue: you're using mem-free firmware, which means the HCA uses system memory to store big chunks of internal state. If something is going wrong with how the memory is mapped to the HCA (or how the HCA writes to it) then that could cause memory corruption -- possibly tied to posting receives to the hardware as part of the MAD initialization. So it could be a driver bug exposed by the new firmware, or a firmware bug. Is Mellanox following this bug? Maybe they have some idea of how to figure out what the HCA is doing that could crash a system. - R. From phillipwils at gmail.com Fri Mar 27 16:57:24 2009 From: phillipwils at gmail.com (Phillip Wilson) Date: Fri, 27 Mar 2009 16:57:24 -0700 Subject: [ofa-general] ***SPAM*** Re: System crashed while booting Linux (ia64) with three Mellanox HCAs (15b3:6274) In-Reply-To: References: <6e4f44220903251939i68cb4268w1284bbb0552643e7@mail.gmail.com> <6e4f44220903271636x7095767fo92eccd72b8bca42c@mail.gmail.com> Message-ID: <6e4f44220903271657u7314605k79e94f320553f5be@mail.gmail.com> I just notified Mellanox of the issue with the 1.2.490 firmware ... On Fri, Mar 27, 2009 at 4:48 PM, Roland Dreier wrote: >  > I spent the last couple of days retracing my steps.  In my haste, I >  > listed the wrong HCA firmware revision.  It was  firmware 1.2.940 that >  > caused the system to crash while booting to Linux.  I have the mthca >  > driver built into the kernel; it is not a loadable driver.  The system >  > boots fine with the 1.2.0 firmware. > > Oh, it's mthca firmware version dependent?  That's a big clue: you're > using mem-free firmware, which means the HCA uses system memory to store > big chunks of internal state.  If something is going wrong with how the > memory is mapped to the HCA (or how the HCA writes to it) then that > could cause memory corruption -- possibly tied to posting receives to > the hardware as part of the MAD initialization. > > So it could be a driver bug exposed by the new firmware, or a firmware bug. > > Is Mellanox following this bug?  Maybe they have some idea of how to > figure out what the HCA is doing that could crash a system. > >  - R. > From vlad at lists.openfabrics.org Sat Mar 28 03:22:25 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sat, 28 Mar 2009 03:22:25 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090328-0200 daily build status Message-ID: <20090328102225.DEE5BE6118C@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From hal.rosenstock at gmail.com Sat Mar 28 04:13:29 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Sat, 28 Mar 2009 07:13:29 -0400 Subject: [ofa-general] OpenSM initialization error In-Reply-To: References: Message-ID: On Fri, Mar 27, 2009 at 6:20 PM, Yicheng Jia wrote: > >> That's max'd out so should be cleared to see if it is continually >> incrementing. > > The number doesn't change after the subnet is up. It's an 16 bit counter which is max'd out and IBTA counters are sticky so once it gets to 65535 it stops unless it's cleared. Did you clear it ? >>Get the portinfo on the SM peer port. Something like: > >>smpquery -D portinfo 0,1 > >>assuming you are doing this from the SM node. > > What if I am running it on a node other than SM node? I don't have tools to > run on SM node now. Are you familiar with directed routes ? If not, you should read that part of the IBA spec (vol 1 14.2.2). >> So different cables are in use ? Same HCA for OpenSM though ? > > The cables are all the same type. HCAs are all same as well. I'm asking whether in the different occurences of this issue (with Cisco and QLogic switches), the same cable and HCA were used for the OpenSM node or whether these are totally separate configurations. -- Hal > Thanks! > > Yicheng Jia From hal.rosenstock at gmail.com Sat Mar 28 05:58:52 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Sat, 28 Mar 2009 08:58:52 -0400 Subject: ***SPAM*** Re: ***SPAM*** Re: [ofa-general] OpenSM and trap 128. In-Reply-To: References: <49CB7723.9080104@ext.bull.net> Message-ID: On Thu, Mar 26, 2009 at 10:48 AM, Hal Rosenstock wrote: > On Thu, Mar 26, 2009 at 8:37 AM, Nicolas Morey Chaisemartin > wrote: >> I was thinking about a solution: >> When receiving a 128 trap (and it triggers a heavy sweep) we check the faulty GUID, lid or port guid. >> If last heavy sweep was triggered by the same faulty port, we wait twice last the last waiting time before forcing the new heavy sweep. >> If it's another source or another reason, we force the heavy sweep right then and set the waiting time to 0. >> >> This way, different problem will still trigger a heavy sweep asap but if only one faulty links triggers it it'll sweep less and less often as it is pretty useless. for this particular case where there are single port link state changes. >> It should solve this case but there may still be a problem when more ports have the same problem... The downside would be that this would obscure "valid" changes on any other ports of the same switches where "simultaneously" occuring on one or more ports. Is that a good tradeoff ? >> Any idea on a way to manage this? >> An ignore mask on traps? (ignore traps for 1 specific problem for x seconds if they happen to often) I think this (trap rate flooding) issue only applies to 4 physically related traps (128-131). -- Hal From marcin.slusarz at gmail.com Sat Mar 28 12:12:08 2009 From: marcin.slusarz at gmail.com (Marcin Slusarz) Date: Sat, 28 Mar 2009 20:12:08 +0100 Subject: [ofa-general] ***SPAM*** Re: Dereferencing freed memory bugs In-Reply-To: References: Message-ID: <49CE7688.2020501@gmail.com> Dan Carpenter wrote: > I added a check to smatch (http://repo.or.cz/w/smatch.git/) to check > for when we dereference > freed memory. > > drivers/dma/dmatest.c +410 dmatest_exit(7) 'dtc' > drivers/dma/dmatest.c +412 dmatest_exit(9) 'dtc' Seems to be fixed by 7cbd4877e5b167b56a3d6033b926a9f925186e12: dmatest: fix use after free in dmatest_exit > drivers/infiniband/hw/nes/nes_cm.c +563 nes_cm_timer_tick(121) 'cm_node' > drivers/infiniband/hw/nes/nes_cm.c +621 nes_cm_timer_tick(179) 'cm_node' > (...) > drivers/usb/host/ehci-hcd.c +1661 itd_complete(79) 'stream' > drivers/usb/host/ehci-hcd.c +2036 sitd_complete(64) 'stream' > drivers/uwb/reset.c +193 __uwb_rc_cmd(26) 'cmd' > (...) > net/netfilter/xt_recent.c +273 recent_mt(69) 'e' > (...) > drivers/media/video/cpia_pp.c +777 cpia_pp_detach(28) 'cpia' > (...) These are less obvious. Adding CCs. Please leave only one of openfabrics/linux-usb/netdev/linux-media in CCs when responding. ps: [s]itd_complete is in drivers/usb/host/ehci-sched.c From ogerlitz at voltaire.com Sat Mar 28 23:44:47 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Sun, 29 Mar 2009 09:44:47 +0300 Subject: [ofa-general] [PATCH] IB/mlx4: Use pgprot_writecombine() for BlueFlame pages In-Reply-To: References: Message-ID: <49CF18DF.8080807@voltaire.com> Roland Dreier wrote: > The PAT work on x86 has finally made pgprot_writecombine() a usable API > for modular drivers. As the comment indicates, this is exactly what we > want to use in mlx4_ib to map BlueFlame pages up to userspace, since > using WC for these pages improves small message latency significantly. > > Signed-off-by: Roland Dreier > --- > Could someone test this patch on top of 2.6.29, and see if it provides > the right latency benefit? If it looks good, I'll send it upstream for 2.6.30. Roland, Can you elaborate a little on the expected latency benefit? If we see today 1us half rtt for small messages, with this patch it should go below? be the same for larger messages? anything else? Or. From monis at Voltaire.COM Sun Mar 29 02:07:09 2009 From: monis at Voltaire.COM (Moni Shoua) Date: Sun, 29 Mar 2009 12:07:09 +0300 Subject: [ofa-general] Re: [ewg] Re: [PATCH] IB/ipoib: set neigh->dgid upon ipoib_neigh creation In-Reply-To: <20090326075135.GA31246@mtls03> References: <20090325145114.GA31206@mtls03> <49CA83F3.9040204@Voltaire.COM> <20090326075135.GA31246@mtls03> Message-ID: <49CF3A3D.6060309@Voltaire.COM> Eli Cohen wrote: > On Wed, Mar 25, 2009 at 09:20:19PM +0200, Yossi Etigin wrote: >> Wouldn't this patch break the fast bond+SM failover recovery? >> >> Up till now, in such case neigh->dgid was the old gid until path record >> query was successful, and that caused to retry the path query until it's >> successful. With that patch, a new path query will not be issued until >> the next arp refresh because neigh->dgid will be set to a valid value >> right after the first packet. > Looks to me like > >> How about keeping the memcpy in path_rec-completion, and also adding it >> to ib_mcast_send (where ipoib neigh is created)? >> > I did not like the idea that a unicast ipoib_neigh can be created and > destroyed till the path is resolved so I preferred to initialize at > creation time. How about the following patch to solve the problem of > failure to resolve a path: > > >>From 4e6e3dd0186803f7547f5e4c233d7525dfcdaec6 Mon Sep 17 00:00:00 2001 > From: Eli Cohen > Date: Wed, 25 Mar 2009 16:22:28 +0200 > Subject: [PATCH] IB/ipoib: set neigh->dgid upon ipoib_neigh creation > > If we fail to do that, there can be superfluous calls to > ipoib_neigh_free()/ipoib_neigh_alloc() due to the following if statement in > ipoib_start_xmit(): > > if (unlikely((memcmp(&neigh->dgid.raw, > skb->dst->neighbour->ha + 4, > sizeof(union ib_gid))) || > > In the case of a unicast GID, it can happen until the path is resolved and is > not so severe. In the case of a multicast address, neigh->dgid is never even > updated, so the corresponding ipoib_neigh will be destroyed and created on > every multicast packet sent. > > Signed-off-by: Eli Cohen > --- > drivers/infiniband/ulp/ipoib/ipoib_main.c | 7 ++++--- > 1 files changed, 4 insertions(+), 3 deletions(-) > > diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c > index 0bd2a4f..b680483 100644 > --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c > +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c > @@ -460,8 +460,6 @@ static void path_rec_completion(int status, > } > kref_get(&path->ah->ref); > neigh->ah = path->ah; > - memcpy(&neigh->dgid.raw, &path->pathrec.dgid.raw, > - sizeof(union ib_gid)); > > if (ipoib_cm_enabled(dev, neigh->neighbour)) { > if (!ipoib_cm_get(neigh)) > @@ -481,7 +479,9 @@ static void path_rec_completion(int status, > __skb_queue_tail(&skqueue, skb); > } > path->valid = 1; > - } > + } else > + list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) > + memset(&neigh->dgid, 0, sizeof neigh->dgid); > > path->query = NULL; > complete(&path->done); > @@ -878,6 +878,7 @@ struct ipoib_neigh *ipoib_neigh_alloc(struct neighbour *neighbour, > neigh->neighbour = neighbour; > neigh->dev = dev; > *to_ipoib_neigh(neighbour) = neigh; > + memcpy(neigh->dgid.raw, neighbour->ha + 4, 16); > skb_queue_head_init(&neigh->queue); > ipoib_cm_set(neigh, NULL); > This might work but still, I don't understand what this patch fixes. What was the initial problem? I agree with Yossi's early comments that the patch breaks the fast fail over we've worked on in the past. From vlad at lists.openfabrics.org Sun Mar 29 03:23:23 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Sun, 29 Mar 2009 03:23:23 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090329-0200 daily build status Message-ID: <20090329102324.2907BE60DCF@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From eli at dev.mellanox.co.il Sun Mar 29 04:48:51 2009 From: eli at dev.mellanox.co.il (Eli Cohen) Date: Sun, 29 Mar 2009 14:48:51 +0300 Subject: [ofa-general] Re: [ewg] Re: [PATCH] IB/ipoib: set neigh->dgid upon ipoib_neigh creation In-Reply-To: <49CF3A3D.6060309@Voltaire.COM> References: <20090325145114.GA31206@mtls03> <49CA83F3.9040204@Voltaire.COM> <20090326075135.GA31246@mtls03> <49CF3A3D.6060309@Voltaire.COM> Message-ID: <20090329114851.GA22847@mtls03> On Sun, Mar 29, 2009 at 12:07:09PM +0300, Moni Shoua wrote: > Eli Cohen wrote: > > On Wed, Mar 25, 2009 at 09:20:19PM +0200, Yossi Etigin wrote: > >> Wouldn't this patch break the fast bond+SM failover recovery? > >> > >> Up till now, in such case neigh->dgid was the old gid until path record > >> query was successful, and that caused to retry the path query until it's > >> successful. With that patch, a new path query will not be issued until > >> the next arp refresh because neigh->dgid will be set to a valid value > >> right after the first packet. > > Looks to me like > > > >> How about keeping the memcpy in path_rec-completion, and also adding it > >> to ib_mcast_send (where ipoib neigh is created)? > >> > > I did not like the idea that a unicast ipoib_neigh can be created and > > destroyed till the path is resolved so I preferred to initialize at > > creation time. How about the following patch to solve the problem of > > failure to resolve a path: > > > > > >>From 4e6e3dd0186803f7547f5e4c233d7525dfcdaec6 Mon Sep 17 00:00:00 2001 > > From: Eli Cohen > > Date: Wed, 25 Mar 2009 16:22:28 +0200 > > Subject: [PATCH] IB/ipoib: set neigh->dgid upon ipoib_neigh creation > > > > If we fail to do that, there can be superfluous calls to > > ipoib_neigh_free()/ipoib_neigh_alloc() due to the following if statement in > > ipoib_start_xmit(): > > > > if (unlikely((memcmp(&neigh->dgid.raw, > > skb->dst->neighbour->ha + 4, > > sizeof(union ib_gid))) || > > > > In the case of a unicast GID, it can happen until the path is resolved and is > > not so severe. In the case of a multicast address, neigh->dgid is never even > > updated, so the corresponding ipoib_neigh will be destroyed and created on > > every multicast packet sent. > > > > Signed-off-by: Eli Cohen > > --- > > drivers/infiniband/ulp/ipoib/ipoib_main.c | 7 ++++--- > > 1 files changed, 4 insertions(+), 3 deletions(-) > > > > diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c > > index 0bd2a4f..b680483 100644 > > --- a/drivers/infiniband/ulp/ipoib/ipoib_main.c > > +++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c > > @@ -460,8 +460,6 @@ static void path_rec_completion(int status, > > } > > kref_get(&path->ah->ref); > > neigh->ah = path->ah; > > - memcpy(&neigh->dgid.raw, &path->pathrec.dgid.raw, > > - sizeof(union ib_gid)); > > > > if (ipoib_cm_enabled(dev, neigh->neighbour)) { > > if (!ipoib_cm_get(neigh)) > > @@ -481,7 +479,9 @@ static void path_rec_completion(int status, > > __skb_queue_tail(&skqueue, skb); > > } > > path->valid = 1; > > - } > > + } else > > + list_for_each_entry_safe(neigh, tn, &path->neigh_list, list) > > + memset(&neigh->dgid, 0, sizeof neigh->dgid); > > > > path->query = NULL; > > complete(&path->done); > > @@ -878,6 +878,7 @@ struct ipoib_neigh *ipoib_neigh_alloc(struct neighbour *neighbour, > > neigh->neighbour = neighbour; > > neigh->dev = dev; > > *to_ipoib_neigh(neighbour) = neigh; > > + memcpy(neigh->dgid.raw, neighbour->ha + 4, 16); > > skb_queue_head_init(&neigh->queue); > > ipoib_cm_set(neigh, NULL); > > > > This might work but still, I don't understand what this patch fixes. What was the initial problem? The problem was that for multicast sends, the corresponding ipoib_neigh would be created and destroyed for every packet because this if will fail: if (unlikely((memcmp(&neigh->dgid.raw, skb->dst->neighbour->ha + 4, sizeof(union ib_gid))) || There is a less severe problem with unicast flow where the ipoib_neigh object might be created and destroyed until the path is resolved. > I agree with Yossi's early comments that the patch breaks the fast fail over we've worked on in the past. From dorfman.eli at gmail.com Sun Mar 29 05:18:31 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Sun, 29 Mar 2009 15:18:31 +0300 Subject: ***SPAM*** Re: [ofa-general] OpenSM and trap 128. In-Reply-To: <49CB7723.9080104@ext.bull.net> References: <49CB7723.9080104@ext.bull.net> Message-ID: <49CF6717.8010904@gmail.com> Nicolas Morey Chaisemartin wrote: > Hi, > > We've noticed while setting up a new cluster a problem with OpenSM. > As usual, there are some cable problems while plugging the cluster but one of the cable was changing state over 10 000 thousands times per second (OFF/ON) and sending each time a 128 trap to OpenSM. > Therefore, OpenSM is constantly rerouting the whole interconnect (every 10s or so). > Fixing the cable will solve our problem, but I still think something should be done about this. > > Though OpenSM behaviour was OK, it was really difficult to find where the performances problems came from. > All our diagnostics tools (mostly using infiniband diags) were failing to see the problem. > Infiniband diags commands fail toward the faulty port but it was hard to say if port was faulty or if it was due to high load on the SM and dropped VL15 messages. > > I was thinking about a solution: > When receiving a 128 trap (and it triggers a heavy sweep) we check the faulty GUID, lid or port guid. > If last heavy sweep was triggered by the same faulty port, we wait twice last the last waiting time before forcing the new heavy sweep. > If it's another source or another reason, we force the heavy sweep right then and set the waiting time to 0. Note that trap 128 is generated by a switch while reporting that one of his ports has changed. The changed port GUID/LID is not reported in the trap. You can change sweep_on_trap option in opensm.conf to FALSE. This should stop opensm heavy sweeps. > > This way, different problem will still trigger a heavy sweep asap but if only one faulty links triggers it it'll sweep less and less often as it is pretty useless. > > It should solve this case but there may still be a problem when more ports have the same problem... > > Any idea on a way to manage this? > An ignore mask on traps? (ignore traps for 1 specific problem for x seconds if they happen to often) > > Thanks > > Nicolas > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From rdreier at cisco.com Sun Mar 29 09:35:44 2009 From: rdreier at cisco.com (Roland Dreier) Date: Sun, 29 Mar 2009 09:35:44 -0700 Subject: [ofa-general] [PATCH] IB/mlx4: Use pgprot_writecombine() for BlueFlame pages In-Reply-To: <49CF18DF.8080807@voltaire.com> (Or Gerlitz's message of "Sun, 29 Mar 2009 09:44:47 +0300") References: <49CF18DF.8080807@voltaire.com> Message-ID: > Can you elaborate a little on the expected latency benefit? If we see > today 1us half rtt for small messages, with this patch it should go > below? be the same for larger messages? anything else? This should bring mainline kernel small message latency to the same level that OFED gets with the PAT support it hacks in. - R. From or.gerlitz at gmail.com Sun Mar 29 10:06:38 2009 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Sun, 29 Mar 2009 20:06:38 +0300 Subject: ***SPAM*** Re: [ofa-general] [PATCH] IB/mlx4: Use pgprot_writecombine() for BlueFlame pages In-Reply-To: References: <49CF18DF.8080807@voltaire.com> Message-ID: <15ddcffd0903291006g4b7549cfj1879dd67518f8bff@mail.gmail.com> On Sun, Mar 29, 2009 at 7:35 PM, Roland Dreier wrote: > This should bring mainline kernel small message latency to the same > level that OFED gets with the PAT support it hacks in. Interesting... so the ofed support for blue flame (we are talking on PIO, correct?) is a hack that magically works even on 2.6.18 et al older kernels? does it have any side effects? Or. From swise at opengridcomputing.com Sun Mar 29 10:47:09 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Sun, 29 Mar 2009 12:47:09 -0500 Subject: [ofa-general] [PATCH 2.6.30] RDMA/cxgb3: Release dependent resources only when endpoint memory is freed. Message-ID: <20090329174709.21018.42547.stgit@build.ogc.int> The cxgb3 l2t entry, hwtid, and dst entry were being released before all the iwch_ep references were released. This can cause a crash in t3_l2t_send_slow() and other places where the l2t entry is used. The fix is to defer releasing these resources until all endpoint references are gone. Details: - move flags field to the iwch_ep_common struct. - add a flag indicating resources are to be released. - release resources at endpoint free time instead of close/abort time. Signed-off-by: Steve Wise --- drivers/infiniband/hw/cxgb3/iwch_cm.c | 26 +++++++++++++++----------- drivers/infiniband/hw/cxgb3/iwch_cm.h | 3 ++- 2 files changed, 17 insertions(+), 12 deletions(-) diff --git a/drivers/infiniband/hw/cxgb3/iwch_cm.c b/drivers/infiniband/hw/cxgb3/iwch_cm.c index ad38c45..042cc4d 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_cm.c +++ b/drivers/infiniband/hw/cxgb3/iwch_cm.c @@ -282,18 +282,22 @@ static void *alloc_ep(int size, gfp_t gfp) void __free_ep(struct kref *kref) { - struct iwch_ep_common *epc; - epc = container_of(kref, struct iwch_ep_common, kref); - PDBG("%s ep %p state %s\n", __func__, epc, states[state_read(epc)]); - kfree(epc); + struct iwch_ep *ep; + ep = container_of(container_of(kref, struct iwch_ep_common, kref), + struct iwch_ep, com); + PDBG("%s ep %p state %s\n", __func__, ep, states[state_read(&ep->com)]); + if (ep->com.flags & RELEASE_RESOURCES) { + cxgb3_remove_tid(ep->com.tdev, (void *)ep, ep->hwtid); + dst_release(ep->dst); + l2t_release(L2DATA(ep->com.tdev), ep->l2t); + } + kfree(ep); } static void release_ep_resources(struct iwch_ep *ep) { PDBG("%s ep %p tid %d\n", __func__, ep, ep->hwtid); - cxgb3_remove_tid(ep->com.tdev, (void *)ep, ep->hwtid); - dst_release(ep->dst); - l2t_release(L2DATA(ep->com.tdev), ep->l2t); + ep->com.flags |= RELEASE_RESOURCES; put_ep(&ep->com); } @@ -1152,8 +1156,8 @@ static int abort_rpl(struct t3cdev *tdev, struct sk_buff *skb, void *ctx) * We get 2 abort replies from the HW. The first one must * be ignored except for scribbling that we need one more. */ - if (!(ep->flags & ABORT_REQ_IN_PROGRESS)) { - ep->flags |= ABORT_REQ_IN_PROGRESS; + if (!(ep->com.flags & ABORT_REQ_IN_PROGRESS)) { + ep->com.flags |= ABORT_REQ_IN_PROGRESS; return CPL_RET_BUF_DONE; } @@ -1557,8 +1561,8 @@ static int peer_abort(struct t3cdev *tdev, struct sk_buff *skb, void *ctx) * We get 2 peer aborts from the HW. The first one must * be ignored except for scribbling that we need one more. */ - if (!(ep->flags & PEER_ABORT_IN_PROGRESS)) { - ep->flags |= PEER_ABORT_IN_PROGRESS; + if (!(ep->com.flags & PEER_ABORT_IN_PROGRESS)) { + ep->com.flags |= PEER_ABORT_IN_PROGRESS; return CPL_RET_BUF_DONE; } diff --git a/drivers/infiniband/hw/cxgb3/iwch_cm.h b/drivers/infiniband/hw/cxgb3/iwch_cm.h index d7c7e09..43c0aea 100644 --- a/drivers/infiniband/hw/cxgb3/iwch_cm.h +++ b/drivers/infiniband/hw/cxgb3/iwch_cm.h @@ -147,6 +147,7 @@ enum iwch_ep_state { enum iwch_ep_flags { PEER_ABORT_IN_PROGRESS = (1 << 0), ABORT_REQ_IN_PROGRESS = (1 << 1), + RELEASE_RESOURCES = (1 << 2), }; struct iwch_ep_common { @@ -161,6 +162,7 @@ struct iwch_ep_common { wait_queue_head_t waitq; int rpl_done; int rpl_err; + u32 flags; }; struct iwch_listen_ep { @@ -188,7 +190,6 @@ struct iwch_ep { u16 plen; u32 ird; u32 ord; - u32 flags; }; static inline struct iwch_ep *to_ep(struct iw_cm_id *cm_id) From jackm at dev.mellanox.co.il Mon Mar 30 00:06:29 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 30 Mar 2009 10:06:29 +0300 Subject: [ofa-general] [PATCH] IB/mlx4: Use pgprot_writecombine() for BlueFlame pages In-Reply-To: References: Message-ID: <200903301006.30442.jackm@dev.mellanox.co.il> On Saturday 28 March 2009 01:15, Roland Dreier wrote: > -               vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot); > +               vma->vm_page_prot = pgprot_writecombine(vma->vm_page_prot); > Roland, I notice in the 2.6.29 code that: 1. There is a function validate_pat_support() that should be called to make sure that pat support is available. Should the driver call this function? (the linux-cross-referencer did not find any callers). 2. What happens with non-X86 architectures? The function pgprot_writecombine() is defined only in X86 archs. - Jack From ogerlitz at Voltaire.com Mon Mar 30 00:07:26 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Mon, 30 Mar 2009 10:07:26 +0300 Subject: [ofa-general] [PATCH] IB/mlx4: Use pgprot_writecombine() for BlueFlame pages In-Reply-To: References: <49CF18DF.8080807@voltaire.com> Message-ID: <49D06FAE.90907@Voltaire.com> Roland Dreier wrote: > This should bring mainline kernel small message latency to the same > level that OFED gets with the PAT support it hacks in. Doing some quick benchmarking today, I couldn't see a notable latency difference before and after the patch, but maybe Yosi would be able to dig deeper and find something... Or. From jackm at dev.mellanox.co.il Mon Mar 30 01:17:31 2009 From: jackm at dev.mellanox.co.il (Jack Morgenstein) Date: Mon, 30 Mar 2009 11:17:31 +0300 Subject: ***SPAM*** Re: [ofa-general] [PATCH] IB/mlx4: Use pgprot_writecombine() for BlueFlame pages In-Reply-To: <15ddcffd0903291006g4b7549cfj1879dd67518f8bff@mail.gmail.com> References: <15ddcffd0903291006g4b7549cfj1879dd67518f8bff@mail.gmail.com> Message-ID: <200903301117.32355.jackm@dev.mellanox.co.il> On Sunday 29 March 2009 20:06, Or Gerlitz wrote: > On Sun, Mar 29, 2009 at 7:35 PM, Roland Dreier wrote: > > This should bring mainline kernel small message latency to the same > > level that OFED gets with the PAT support it hacks in. > > Interesting... so the ofed support for blue flame (we are talking on > PIO, correct?) is a hack that magically works even on 2.6.18 et al > older kernels? does it have any side effects? > > Or. The problem was that it was not standardized for all drivers (as a set of kernel procedures available for all). The hack worked by replacing one of the entries (modes) in the PAT with the WC (write combining) mode, and then allocating memory with pgprot_wc() to set the PAT reference to the WC PAT. The mlx4 driver does the PAT replacement at driver start, and undoes the replacement at driver unload. However, if other device drivers use the same hack, then things get messed up -- the driver unload creates problems for these other drivers (or their unload creates problems for us). You can see the write combining driver code in file: kernel_patches/fixes/mlx4_0010_add_wc.patch - Jack From vlad at lists.openfabrics.org Mon Mar 30 03:24:26 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Mon, 30 Mar 2009 03:24:26 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090330-0200 daily build status Message-ID: <20090330102426.B5060E60DC0@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From hal.rosenstock at gmail.com Mon Mar 30 04:47:48 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Mon, 30 Mar 2009 07:47:48 -0400 Subject: ***SPAM*** Re: ***SPAM*** Re: [ofa-general] OpenSM and trap 128. In-Reply-To: <49CF6717.8010904@gmail.com> References: <49CB7723.9080104@ext.bull.net> <49CF6717.8010904@gmail.com> Message-ID: On Sun, Mar 29, 2009 at 8:18 AM, Eli Dorfman (Voltaire) wrote: > You can change sweep_on_trap option in opensm.conf to FALSE. > This should stop opensm heavy sweeps. Yes but is that what is really desired ? If sweep on trap is disabled, there is no reaction to legitimate subnet changes. -- Hal From dorons at Voltaire.COM Mon Mar 30 05:08:57 2009 From: dorons at Voltaire.COM (Doron Shoham) Date: Mon, 30 Mar 2009 15:08:57 +0300 Subject: [ofa-general] [PATCH] ibsim: support xmitwait counters Message-ID: <49D0B659.2040107@Voltaire.COM> support xmitwait counters Signed-off-by: Doron Shoham --- ibsim/sim.h | 3 +++ ibsim/sim_mad.c | 11 ++++++++++- 2 files changed, 13 insertions(+), 1 deletions(-) diff --git a/ibsim/sim.h b/ibsim/sim.h index 5e9b4f0..5a8a92f 100644 --- a/ibsim/sim.h +++ b/ibsim/sim.h @@ -111,6 +111,7 @@ enum GS_PERF_COUNTER_SELECT_MASK { GS_PERF_RCV_BYTES_MASK = (1UL << 13), // PORT_RCV_DATA GS_PERF_XMT_PKTS_MASK = (1UL << 14), // PORT_XMIT_PKTS GS_PERF_RCV_PKTS_MASK = (1UL << 15), // PORT_RCV_PKTS + GS_PERF_XMT_WAIT_MASK = (1UL << 16), // PORT_XMIT_WAIT }; enum GS_PC_EXT_SELECT_MASK { @@ -141,6 +142,7 @@ enum GS_PERF_COUNTER_SELECT_LIMIT { GS_PERF_RCV_BYTES_LIMIT = 0xffffffff, GS_PERF_XMT_PKTS_LIMIT = 0xffffffff, GS_PERF_RCV_PKTS_LIMIT = 0xffffffff, + GS_PERF_XMT_WAIT_LIMIT = 0xffffffff, }; typedef struct Port Port; @@ -182,6 +184,7 @@ struct Portcounters { uint8_t errs_rcvconstraint; uint16_t errs_rcvswitchrelay; uint8_t errs_excessbufovrrun; + uint32_t xmt_wait; }; struct Port { diff --git a/ibsim/sim_mad.c b/ibsim/sim_mad.c index d2c313c..f4e7359 100644 --- a/ibsim/sim_mad.c +++ b/ibsim/sim_mad.c @@ -567,6 +567,8 @@ static void pc_reset(Portcounters * pc, uint mask) pc->flow_xmt_pkts = 0; if (mask & GS_PERF_RCV_PKTS_MASK) pc->flow_rcv_pkts = 0; + if (mask & GS_PERF_XMT_WAIT_MASK) + pc->xmt_wait = 0; } static inline uint32_t addval(uint32_t val, uint32_t delta, uint32_t max) @@ -641,6 +643,9 @@ static int pc_updated(Port ** srcport, Port * destport) static void pc_sum(Portcounters * totals, Portcounters * pc) { + totals->xmt_wait = + addval(totals->xmt_wait, pc->xmt_wait, + GS_PERF_XMT_WAIT_LIMIT); totals->flow_xmt_pkts = addval(totals->flow_xmt_pkts, pc->flow_xmt_pkts, GS_PERF_XMT_PKTS_LIMIT); @@ -691,6 +696,7 @@ static void pc_sum(Portcounters * totals, Portcounters * pc) static void pc_get(Portcounters * pc, uint8_t * data) { + mad_set_field(data, 0, IB_PC_XMT_WAIT_F, pc->xmt_wait); mad_set_field(data, 0, IB_PC_XMT_PKTS_F, pc->flow_xmt_pkts); mad_set_field(data, 0, IB_PC_XMT_BYTES_F, pc->flow_xmt_bytes); mad_set_field(data, 0, IB_PC_RCV_PKTS_F, pc->flow_rcv_pkts); @@ -716,7 +722,7 @@ static int do_portcounters(Port * port, unsigned op, uint32_t unused, Node *node = port->node; int portnum = mad_get_field(data, 0, IB_PC_PORT_SELECT_F); Portcounters totals; - uint mask; + uint mask, mask2; Port *p; int i; @@ -731,6 +737,9 @@ static int do_portcounters(Port * port, unsigned op, uint32_t unused, node->nodeguid, port->portguid, portnum); mask = mad_get_field(data, 0, IB_PC_COUNTER_SELECT_F); + mask2 = mad_get_field(data, 0, IB_PC_COUNTER_SELECT2_F); + if (mask2) + mask |= GS_PERF_XMT_WAIT_MASK; if (portnum != 0xff) { if (!(p = node_get_port(node, portnum))) -- 1.5.5 From rdreier at cisco.com Mon Mar 30 07:14:55 2009 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 30 Mar 2009 07:14:55 -0700 Subject: [ofa-general] [PATCH] IB/mlx4: Use pgprot_writecombine() for BlueFlame pages In-Reply-To: <200903301006.30442.jackm@dev.mellanox.co.il> (Jack Morgenstein's message of "Mon, 30 Mar 2009 10:06:29 +0300") References: <200903301006.30442.jackm@dev.mellanox.co.il> Message-ID: > Roland, I notice in the 2.6.29 code that: > 1. There is a function validate_pat_support() that should be called to > make sure that pat support is available. Should the driver call this function? > (the linux-cross-referencer did not find any callers). validate_pat_support() was removed in the upstream kernel in 75a04811 ("x86: handle PAT more like other CPU features"). So no, the driver shouldn't call it. > 2. What happens with non-X86 architectures? The function pgprot_writecombine() is defined > only in X86 archs. Look at include/asm-generic/pgtable.h: #ifndef pgprot_writecombine #define pgprot_writecombine pgprot_noncached #endif - R. From swise at opengridcomputing.com Mon Mar 30 08:23:28 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Mon, 30 Mar 2009 10:23:28 -0500 Subject: [ofa-general] mlx4 post_send question Message-ID: <49D0E3F0.2020105@opengridcomputing.com> Hey Roland, Quick question: If I the mlx4 post_send function returns an error synchronously, then it will _NOT_ have posted the WR that caused the error, correct? Further, I won't get a CQE for that failed WR. Correct? This is in reference to an frmr problem Vu is seeing. I just want to make sure that synchronous errors to post_send() don't generate completion events. Thanks, Steve. From rdreier at cisco.com Mon Mar 30 08:32:05 2009 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 30 Mar 2009 08:32:05 -0700 Subject: [ofa-general] mlx4 post_send question In-Reply-To: <49D0E3F0.2020105@opengridcomputing.com> (Steve Wise's message of "Mon, 30 Mar 2009 10:23:28 -0500") References: <49D0E3F0.2020105@opengridcomputing.com> Message-ID: > If I the mlx4 post_send function returns an error synchronously, then > it will _NOT_ have posted the WR that caused the error, correct? > Further, I won't get a CQE for that failed WR. Correct? Right, unless there's a bug I don't know about. - R. From rdreier at cisco.com Mon Mar 30 08:38:17 2009 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 30 Mar 2009 08:38:17 -0700 Subject: [ofa-general] [PATCH 2.6.30] RDMA/cxgb3: Release dependent resources only when endpoint memory is freed. In-Reply-To: <20090329174709.21018.42547.stgit@build.ogc.int> (Steve Wise's message of "Sun, 29 Mar 2009 12:47:09 -0500") References: <20090329174709.21018.42547.stgit@build.ogc.int> Message-ID: thanks, applied From rdreier at cisco.com Mon Mar 30 08:38:25 2009 From: rdreier at cisco.com (Roland Dreier) Date: Mon, 30 Mar 2009 08:38:25 -0700 Subject: [ofa-general] [PATCH v3] RDMA/cxgb3: Handle EEH events for active connections. In-Reply-To: <20090309102014.5738.98445.stgit@build.ogc.int> (Steve Wise's message of "Mon, 09 Mar 2009 05:20:14 -0500") References: <20090309102014.5738.98445.stgit@build.ogc.int> Message-ID: thanks, applied From yosefe at Voltaire.COM Mon Mar 30 08:49:37 2009 From: yosefe at Voltaire.COM (Yossi Etigin) Date: Mon, 30 Mar 2009 18:49:37 +0300 Subject: [ofa-general] Re: [PATCH v3] mlx4_ib: Optimize hugetlab pages support In-Reply-To: <20090129152725.GA26284@mtls03> References: <20090129152725.GA26284@mtls03> Message-ID: <49D0EA11.2040409@Voltaire.COM> Eli Cohen wrote: > Since Linux may not merge adjacent pages into a single scatter entry through > calls to dma_map_sg(), we check the special case of hugetlb pages which are > likely to be mapped to coniguous dma addresses and if they are, take advantage > of this. This will result in a significantly lower number of MTT segments used > for registering hugetlb memory regions. > How about the one below - it fixes bugzilla #1569 (fix mapping for size that is not on page boundary): --- Since Linux may not merge adjacent pages into a single scatter entry through calls to dma_map_sg(), we check the special case of hugetlb pages which are likely to be mapped to coniguous dma addresses and if they are, take advantage of this. This will result in a significantly lower number of MTT segments used for registering hugetlb memory regions. Signed-off-by: Eli Cohen --- drivers/infiniband/hw/mlx4/mr.c | 81 ++++++++++++++++++++++++++++++++++---- 1 files changed, 72 insertions(+), 9 deletions(-) Index: b/drivers/infiniband/hw/mlx4/mr.c =================================================================== --- a/drivers/infiniband/hw/mlx4/mr.c 2008-11-19 21:32:15.000000000 +0200 +++ b/drivers/infiniband/hw/mlx4/mr.c 2009-03-30 18:29:55.000000000 +0300 @@ -119,6 +119,70 @@ out: return err; } +static int handle_hugetlb_user_mr(struct ib_pd *pd, struct mlx4_ib_mr *mr, + u64 start, u64 virt_addr, int access_flags) +{ +#if defined(CONFIG_HUGETLB_PAGE) && !defined(__powerpc__) && !defined(__ia64__) + struct mlx4_ib_dev *dev = to_mdev(pd->device); + struct ib_umem_chunk *chunk; + unsigned dsize; + dma_addr_t daddr; + unsigned cur_size = 0; + dma_addr_t uninitialized_var(cur_addr); + int n; + struct ib_umem *umem = mr->umem; + u64 *arr; + int err = 0; + int i; + int j = 0; + int off = start & (HPAGE_SIZE - 1); + + n = DIV_ROUND_UP(off + umem->length, HPAGE_SIZE); + arr = kmalloc(n * sizeof *arr, GFP_KERNEL); + if (!arr) + return -ENOMEM; + + list_for_each_entry(chunk, &umem->chunk_list, list) + for (i = 0; i < chunk->nmap; ++i) { + daddr = sg_dma_address(&chunk->page_list[i]); + dsize = sg_dma_len(&chunk->page_list[i]); + if (!cur_size) { + cur_addr = daddr; + cur_size = dsize; + } else if (cur_addr + cur_size != daddr) { + err = -EINVAL; + goto out; + } else + cur_size += dsize; + + if (cur_size > HPAGE_SIZE) { + err = -EINVAL; + goto out; + } else if (cur_size == HPAGE_SIZE) { + cur_size = 0; + arr[j++] = cur_addr; + } + } + + if (cur_size) { + arr[j++] = cur_addr; + } + + err = mlx4_mr_alloc(dev->dev, to_mpd(pd)->pdn, virt_addr, umem->length, + convert_access(access_flags), n, HPAGE_SHIFT, &mr->mmr); + if (err) + goto out; + + err = mlx4_write_mtt(dev->dev, &mr->mmr.mtt, 0, n, arr); + +out: + kfree(arr); + return err; +#else + return -ENOSYS; +#endif +} + struct ib_mr *mlx4_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length, u64 virt_addr, int access_flags, struct ib_udata *udata) @@ -140,17 +204,20 @@ struct ib_mr *mlx4_ib_reg_user_mr(struct goto err_free; } - n = ib_umem_page_count(mr->umem); - shift = ilog2(mr->umem->page_size); - - err = mlx4_mr_alloc(dev->dev, to_mpd(pd)->pdn, virt_addr, length, - convert_access(access_flags), n, shift, &mr->mmr); - if (err) - goto err_umem; - - err = mlx4_ib_umem_write_mtt(dev, &mr->mmr.mtt, mr->umem); - if (err) - goto err_mr; + if (!mr->umem->hugetlb || + handle_hugetlb_user_mr(pd, mr, start, virt_addr, access_flags)) { + n = ib_umem_page_count(mr->umem); + shift = ilog2(mr->umem->page_size); + + err = mlx4_mr_alloc(dev->dev, to_mpd(pd)->pdn, virt_addr, length, + convert_access(access_flags), n, shift, &mr->mmr); + if (err) + goto err_umem; + + err = mlx4_ib_umem_write_mtt(dev, &mr->mmr.mtt, mr->umem); + if (err) + goto err_mr; + } err = mlx4_mr_enable(dev->dev, &mr->mmr); if (err) From yosefe at Voltaire.COM Mon Mar 30 09:52:37 2009 From: yosefe at Voltaire.COM (Yossi Etigin) Date: Mon, 30 Mar 2009 19:52:37 +0300 Subject: [ofa-general] Re: [PATCH v2] ipoib: fix a deadlock between ipoib start/stop and child interface create/delete In-Reply-To: References: <495B6B8B.50803@Voltaire.COM> Message-ID: <49D0F8D5.1090204@Voltaire.COM> Roland Dreier wrote: > @@ -162,10 +165,11 @@ int ipoib_vlan_delete(struct net_device *pdev, unsigned short pkey) > > ppriv = netdev_priv(pdev); > > + rtnl_lock(); > mutex_lock(&ppriv->vlan_mutex); > list_for_each_entry_safe(priv, tpriv, &ppriv->child_intfs, list) { > if (priv->pkey == pkey) { > - unregister_netdev(priv->dev); > + unregister_netdevice(priv->dev); > ipoib_dev_cleanup(priv->dev); > list_del(&priv->list); > free_netdev(priv->dev); I got a kernel bug and I think it's related to that patch - doing free_netdev() asserts the device state to be NETREG_UNREGISTERED, and unregister_netdev() does not really cause a state change until rtnl is unlocked. kernel BUG at net/core/dev.c:4746! invalid opcode: 0000 [#2] SMP last sysfs file: /sys/class/net/ib0/delete_child CPU 1 Modules linked in: rdma_ucm ib_ucm rdma_cm iw_cm ib_addr ib_ipoib ib_cm ib_sa inet_lro ib_uverbs ib_umad mlx4_ib mlx4_core ib_mthca ib_mad ib_core nfs lockd nfs_acl auth_rpcgss sunrpc bonding af_packet ipv6 button battery ac loop dm_mod tg3 libphy forcedeth parport_pc lp parport ext3 jbd edd fan thermal processor thermal_sys amd74xx ide_gd_mod ide_core [last unloaded: mlx4_core] Pid: 15079, comm: bash Tainted: G D 2.6.29-rc2-smp #23 RIP: 0010:[] [] free_netdev+0x64/0x81 RSP: 0000:ffff8800ba82be48 EFLAGS: 00010297 RAX: 0000000000000002 RBX: ffff8800b98b2000 RCX: ffff8800b98b2060 RDX: ffff8800b98b2060 RSI: 0000000000000003 RDI: ffff8800b98b2028 RBP: ffff8800b98b2028 R08: ffff8800b98b2000 R09: ffff880001101c80 R10: 0000000000000002 R11: 0000000000000000 R12: ffff8800c7d16700 R13: ffff8800c7d16770 R14: 000000000000faaa R15: ffffffff8059e290 FS: 00007f806b4306d0(0000) GS:ffff8800cf663340(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000005a9008 CR3: 00000000b0111000 CR4: 00000000000006e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff4ff0 DR7: 0000000000000400 Process bash (pid: 15079, threadinfo ffff8800ba82a000, task ffff8800cd4e31a0) Stack: ffff8800b98b3088 0000000000000000 ffff8800b98b3088 ffffffffa0222447 ffff8800c7d163e8 0000000000000007 ffff8800cb6bbaf0 00000000ffffffed ffff8800c7d16478 ffffffffa021c1a1 00007f806abf4000 ffff8800ce805cc0 Call Trace: [] ? ipoib_vlan_delete+0x67/0xec [ib_ipoib] [] ? delete_child+0x40/0x5d [ib_ipoib] [] ? sysfs_write_file+0xd2/0x110 [] ? vfs_write+0xad/0x136 [] ? sys_write+0x45/0x6e [] ? system_call_fastpath+0x16/0x1b Code: ed 38 48 39 c2 75 e3 8b 83 b0 03 00 00 85 c0 75 15 0f b7 83 68 01 00 00 48 29 c3 58 48 89 df 5b 5d e9 04 b6 ec ff 83 f8 03 74 04 <0f> 0b eb fe c7 83 b0 03 00 00 04 00 00 00 48 8d bb e8 03 00 00 RIP [] free_netdev+0x64/0x81 RSP ---[ end trace 14a605f15a1101d0 ]--- -- --Yossi From yosefe at Voltaire.COM Mon Mar 30 10:37:56 2009 From: yosefe at Voltaire.COM (Yossi Etigin) Date: Mon, 30 Mar 2009 20:37:56 +0300 Subject: [ofa-general] [PATCH] ipoib: fix free_netdev() BUG while deleting a vlan Message-ID: <49D10374.2090505@Voltaire.COM> free_netdev() should be called after rtnl_lock is released. Failing to to so might trigger an BUG in free_netdev() which asserts that the device state is NETREG_UNREGISTERED. Signed-off-by: Yossi Etigin -- Index: b/drivers/infiniband/ulp/ipoib/ipoib_vlan.c =================================================================== --- a/drivers/infiniband/ulp/ipoib/ipoib_vlan.c 2009-03-10 18:21:47.000000000 +0200 +++ b/drivers/infiniband/ulp/ipoib/ipoib_vlan.c 2009-03-30 20:18:29.000000000 +0300 @@ -147,7 +147,10 @@ register_failed: ipoib_dev_cleanup(priv->dev); device_init_failed: + mutex_unlock(&ppriv->vlan_mutex); + rtnl_unlock(); free_netdev(priv->dev); + return result; err: mutex_unlock(&ppriv->vlan_mutex); @@ -158,7 +161,6 @@ err: int ipoib_vlan_delete(struct net_device *pdev, unsigned short pkey) { struct ipoib_dev_priv *ppriv, *priv, *tpriv; - int ret = -ENOENT; if (!capable(CAP_NET_ADMIN)) return -EPERM; @@ -172,14 +174,15 @@ int ipoib_vlan_delete(struct net_device unregister_netdevice(priv->dev); ipoib_dev_cleanup(priv->dev); list_del(&priv->list); - free_netdev(priv->dev); - ret = 0; - break; + mutex_unlock(&ppriv->vlan_mutex); + rtnl_unlock(); + free_netdev(priv->dev); + return 0; } } mutex_unlock(&ppriv->vlan_mutex); rtnl_unlock(); - return ret; + return -ENOENT; } From yosefe at Voltaire.COM Mon Mar 30 10:50:43 2009 From: yosefe at Voltaire.COM (Yossi Etigin) Date: Mon, 30 Mar 2009 20:50:43 +0300 Subject: [ofa-general] [PATCH v2] rdma_cm: create cm id even when port is down In-Reply-To: <49D0E1A4.3070804@Voltaire.COM> References: <49D0E1A4.3070804@Voltaire.COM> Message-ID: <49D10673.3000006@Voltaire.COM> When doing rdma_resolve_addr() and relevant port is down, the function fails and rdma_cm id is not bound to the device. Therefore, application does not have device handle and cannot wait for the port to become active. The function fails because ipoib is not joined to the multicast group and therefore sa does not have a multicast record to take a qkey from. The proposed patch is to make lazy qkey resolution - cma_set_qkey will set id_priv->qkey if it was not set, and will be called just before the qkey is really required. Signed-off-by: Yossi Etigin --- Changes from v1: - fix a missing return value in cma_set_qkey() drivers/infiniband/core/cma.c | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-) Index: b/drivers/infiniband/core/cma.c =================================================================== --- a/drivers/infiniband/core/cma.c 2009-03-30 18:27:36.000000000 +0300 +++ b/drivers/infiniband/core/cma.c 2009-03-30 19:01:30.000000000 +0300 @@ -297,21 +297,25 @@ static void cma_detach_from_dev(struct r id_priv->cma_dev = NULL; } -static int cma_set_qkey(struct ib_device *device, u8 port_num, - enum rdma_port_space ps, - struct rdma_dev_addr *dev_addr, u32 *qkey) +static int cma_set_qkey(struct rdma_id_private *id_priv) { struct ib_sa_mcmember_rec rec; int ret = 0; - switch (ps) { + if (id_priv->qkey) + return 0; + + switch (id_priv->id.ps) { case RDMA_PS_UDP: - *qkey = RDMA_UDP_QKEY; + id_priv->qkey = RDMA_UDP_QKEY; break; case RDMA_PS_IPOIB: - ib_addr_get_mgid(dev_addr, &rec.mgid); - ret = ib_sa_get_mcmember_rec(device, port_num, &rec.mgid, &rec); - *qkey = be32_to_cpu(rec.qkey); + ib_addr_get_mgid(&id_priv->id.route.addr.dev_addr, &rec.mgid); + ret = ib_sa_get_mcmember_rec(id_priv->id.device, + id_priv->id.port_num, &rec.mgid, + &rec); + if (!ret) + id_priv->qkey = be32_to_cpu(rec.qkey); break; default: break; @@ -341,12 +345,7 @@ static int cma_acquire_dev(struct rdma_i ret = ib_find_cached_gid(cma_dev->device, &gid, &id_priv->id.port_num, NULL); if (!ret) { - ret = cma_set_qkey(cma_dev->device, - id_priv->id.port_num, - id_priv->id.ps, dev_addr, - &id_priv->qkey); - if (!ret) - cma_attach_to_dev(id_priv, cma_dev); + cma_attach_to_dev(id_priv, cma_dev); break; } } @@ -578,6 +577,10 @@ static int cma_ib_init_qp_attr(struct rd *qp_attr_mask = IB_QP_STATE | IB_QP_PKEY_INDEX | IB_QP_PORT; if (cma_is_ud_ps(id_priv->id.ps)) { + ret = cma_set_qkey(id_priv); + if (ret) + return ret; + qp_attr->qkey = id_priv->qkey; *qp_attr_mask |= IB_QP_QKEY; } else { @@ -2201,6 +2204,12 @@ static int cma_sidr_rep_handler(struct i event.status = ib_event->param.sidr_rep_rcvd.status; break; } + ret = cma_set_qkey(id_priv); + if (ret) { + event.event = RDMA_CM_EVENT_ADDR_ERROR; + event.status = -EINVAL; + break; + } if (id_priv->qkey != rep->qkey) { event.event = RDMA_CM_EVENT_UNREACHABLE; event.status = -EINVAL; @@ -2480,10 +2489,14 @@ static int cma_send_sidr_rep(struct rdma const void *private_data, int private_data_len) { struct ib_cm_sidr_rep_param rep; + int ret; memset(&rep, 0, sizeof rep); rep.status = status; if (status == IB_SIDR_SUCCESS) { + ret = cma_set_qkey(id_priv); + if (ret) + return ret; rep.qp_num = id_priv->qp_num; rep.qkey = id_priv->qkey; } _______________________________________________ ewg mailing list ewg at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg From yosefe at Voltaire.COM Mon Mar 30 11:48:58 2009 From: yosefe at Voltaire.COM (Yossi Etigin) Date: Mon, 30 Mar 2009 21:48:58 +0300 Subject: [ofa-general] [PATCH] rdma_cm: Use rate from ipoib broadcast when joining ipoib multicast Message-ID: <49D1141A.40700@Voltaire.COM> When joining IPoIB multicast group, use the same rate as in the broadcast group. Otherwise, if rdma_cm creates this group before IPoIB does, it might get a different rate. This will cause IPoIB to fail joining to the same group later on, because IPoIB has a strict rate selection. Signed-off-by: Yossi Etigin -- Index: b/drivers/infiniband/core/cma.c =================================================================== --- a/drivers/infiniband/core/cma.c 2009-03-30 19:53:19.000000000 +0300 +++ b/drivers/infiniband/core/cma.c 2009-03-30 21:38:02.000000000 +0300 @@ -2726,6 +2726,10 @@ static int cma_join_ib_multicast(struct IB_SA_MCMEMBER_REC_FLOW_LABEL | IB_SA_MCMEMBER_REC_TRAFFIC_CLASS; + if (id_priv->id.ps == RDMA_PS_IPOIB) + comp_mask |= IB_SA_MCMEMBER_REC_RATE | + IB_SA_MCMEMBER_REC_RATE_SELECTOR; + mc->multicast.ib = ib_sa_join_multicast(&sa_client, id_priv->id.device, id_priv->id.port_num, &rec, comp_mask, GFP_KERNEL, From sean.hefty at intel.com Mon Mar 30 12:03:41 2009 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 30 Mar 2009 12:03:41 -0700 Subject: [ofa-general] RE: [PATCH] rdma_cm: Use rate from ipoib broadcast when joining ipoib multicast In-Reply-To: <49D1141A.40700@Voltaire.COM> References: <49D1141A.40700@Voltaire.COM> Message-ID: > When joining IPoIB multicast group, use the same rate as in the broadcast >group. Otherwise, if rdma_cm creates this group before IPoIB does, it might get >a different rate. This will cause IPoIB to fail joining to the same group later >on, because IPoIB has a strict rate selection. Should the rdma_cm be creating IPoIB multicast groups? From yosefe at Voltaire.COM Mon Mar 30 12:06:19 2009 From: yosefe at Voltaire.COM (Yossi Etigin) Date: Mon, 30 Mar 2009 22:06:19 +0300 Subject: [ofa-general] Re: [PATCH] rdma_cm: Use rate from ipoib broadcast when joining ipoib multicast In-Reply-To: References: <49D1141A.40700@Voltaire.COM> Message-ID: <49D1182B.2020601@Voltaire.COM> Sean Hefty wrote: >> When joining IPoIB multicast group, use the same rate as in the broadcast >> group. Otherwise, if rdma_cm creates this group before IPoIB does, it might get >> a different rate. This will cause IPoIB to fail joining to the same group later >> on, because IPoIB has a strict rate selection. > > Should the rdma_cm be creating IPoIB multicast groups? > Actually I meant the case of using RDMA_PS_IPOIB. From or.gerlitz at gmail.com Mon Mar 30 12:58:26 2009 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Mon, 30 Mar 2009 22:58:26 +0300 Subject: ***SPAM*** Re: [ofa-general] [PATCH] rdma_cm: Use rate from ipoib broadcast when joining ipoib multicast In-Reply-To: <49D1141A.40700@Voltaire.COM> References: <49D1141A.40700@Voltaire.COM> Message-ID: <15ddcffd0903301258m5513e902v186050a22cd96b38@mail.gmail.com> On Mon, Mar 30, 2009 at 9:48 PM, Yossi Etigin wrote: > When joining IPoIB multicast group, use the same rate as in the broadcast > group > + if (id_priv->id.ps == RDMA_PS_IPOIB) > + comp_mask |= IB_SA_MCMEMBER_REC_RATE | > + IB_SA_MCMEMBER_REC_RATE_SELECTOR; > + > you have specified only component mask but not any actual values... Or. -------------- next part -------------- An HTML attachment was scrubbed... URL: From yosefe at voltaire.com Mon Mar 30 13:05:56 2009 From: yosefe at voltaire.com (Yossi Etigin) Date: Mon, 30 Mar 2009 23:05:56 +0300 Subject: [ofa-general] [PATCH] rdma_cm: Use rate from ipoib broadcast when joining ipoib multicast In-Reply-To: <15ddcffd0903301258m5513e902v186050a22cd96b38@mail.gmail.com> References: <49D1141A.40700@Voltaire.COM> <15ddcffd0903301258m5513e902v186050a22cd96b38@mail.gmail.com> Message-ID: <49D12624.2000109@voltaire.com> Or Gerlitz wrote: > On Mon, Mar 30, 2009 at 9:48 PM, Yossi Etigin > wrote: > > When joining IPoIB multicast group, use the same rate as in the > broadcast group > + if (id_priv->id.ps == RDMA_PS_IPOIB) > + comp_mask |= IB_SA_MCMEMBER_REC_RATE | > + IB_SA_MCMEMBER_REC_RATE_SELECTOR; > + > > > you have specified only component mask but not any actual values... > > Or. The values come from ib_sa_get_mcmember_rec( ... , &rec) call earlier. From or.gerlitz at gmail.com Mon Mar 30 13:28:55 2009 From: or.gerlitz at gmail.com (Or Gerlitz) Date: Mon, 30 Mar 2009 23:28:55 +0300 Subject: ***SPAM*** Re: [ofa-general] [PATCH] rdma_cm: Use rate from ipoib broadcast when joining ipoib multicast In-Reply-To: <49D12624.2000109@voltaire.com> References: <49D1141A.40700@Voltaire.COM> <15ddcffd0903301258m5513e902v186050a22cd96b38@mail.gmail.com> <49D12624.2000109@voltaire.com> Message-ID: <15ddcffd0903301328k24271646p3c18a221241ce1fc@mail.gmail.com> On Mon, Mar 30, 2009 at 11:05 PM, Yossi Etigin wrote: > The values come from ib_sa_get_mcmember_rec( ... , &rec) call earlier. of-course, Or. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Ted.Kim at Sun.COM Mon Mar 30 15:54:12 2009 From: Ted.Kim at Sun.COM (Ted H. Kim) Date: Mon, 30 Mar 2009 15:54:12 -0700 Subject: [ofa-general] Sonoma 2009 presentations Message-ID: <49D14D94.50103@sun.com> Folks, Are the Sonoma 2009 presentations online yet? I haven't been able to find them yet on the website. -ted -- Ted H. Kim Sun Microsystems, Inc. ted.kim at sun.com 222 North Sepulveda Blvd., 10th Floor (310) 341-1116 El Segundo, CA 90245 (310) 341-1120 FAX From tziporet at dev.mellanox.co.il Tue Mar 31 02:08:36 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Tue, 31 Mar 2009 12:08:36 +0300 Subject: [ofa-general] ***SPAM*** Re: System crashed while booting Linux (ia64) with three Mellanox HCAs (15b3:6274) In-Reply-To: <6e4f44220903271657u7314605k79e94f320553f5be@mail.gmail.com> References: <6e4f44220903251939i68cb4268w1284bbb0552643e7@mail.gmail.com> <6e4f44220903271636x7095767fo92eccd72b8bca42c@mail.gmail.com> <6e4f44220903271657u7314605k79e94f320553f5be@mail.gmail.com> Message-ID: <49D1DD94.5000500@mellanox.co.il> Phillip Wilson wrote: > I just notified Mellanox of the issue with the 1.2.490 firmware ... > > I will track this in Mellanox Tziporet From tziporet at dev.mellanox.co.il Tue Mar 31 02:47:59 2009 From: tziporet at dev.mellanox.co.il (Tziporet Koren) Date: Tue, 31 Mar 2009 12:47:59 +0300 Subject: [ofa-general] [PATCH] IB/mlx4: Use pgprot_writecombine() for BlueFlame pages In-Reply-To: <49D06FAE.90907@Voltaire.com> References: <49CF18DF.8080807@voltaire.com> <49D06FAE.90907@Voltaire.com> Message-ID: <49D1E6CF.8050101@mellanox.co.il> Or Gerlitz wrote: > > Doing some quick benchmarking today, I couldn't see a notable latency difference before and after the patch, but maybe Yosi would be able to dig deeper and find something... > > Or, The fast path is enabled only when inline in used. Please make sure your test is using inline data Tziporet From vlad at lists.openfabrics.org Tue Mar 31 03:26:03 2009 From: vlad at lists.openfabrics.org (Vladimir Sokolovsky Mellanox) Date: Tue, 31 Mar 2009 03:26:03 -0700 (PDT) Subject: [ofa-general] ofa_1_4_kernel 20090331-0200 daily build status Message-ID: <20090331102603.B5BDAE61156@openfabrics.org> This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_4/linux-2.6.git git_branch: ofed_kernel Common build parameters: Passed: Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16 Passed on x86_64 with linux-2.6.16.21-0.8-smp Passed on x86_64 with linux-2.6.16.43-0.3-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18-1.2798.fc6 Passed on x86_64 with linux-2.6.18-53.el5 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.18-8.el5 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.22.5-31-default Passed on x86_64 with linux-2.6.9-42.ELsmp Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on x86_64 with linux-2.6.9-55.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.16.21-0.8-default Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.25 Passed on ia64 with linux-2.6.26 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.18-8.el5 Passed on ppc64 with linux-2.6.19 Failed: From ogerlitz at voltaire.com Tue Mar 31 04:00:08 2009 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Tue, 31 Mar 2009 14:00:08 +0300 Subject: [ofa-general] [PATCH] IB/mlx4: Use pgprot_writecombine() for BlueFlame pages In-Reply-To: <49D1E6CF.8050101@mellanox.co.il> References: <49CF18DF.8080807@voltaire.com> <49D06FAE.90907@Voltaire.com> <49D1E6CF.8050101@mellanox.co.il> Message-ID: <49D1F7B8.3090004@voltaire.com> > The fast path is enabled only when inline in used. Please make sure > your test is using inline data One of the things I tried was ib_rdma_lat Or. From nicolas.morey-chaisemartin at ext.bull.net Tue Mar 31 04:03:40 2009 From: nicolas.morey-chaisemartin at ext.bull.net (Nicolas Morey Chaisemartin) Date: Tue, 31 Mar 2009 13:03:40 +0200 Subject: [ofa-general] Setting MTU on ib0.????? Message-ID: <49D1F88C.5080501@ext.bull.net> Hi, I saw in openibd script that through creating a /etc/sysconfig/network-script/ifcfg-ib?.???? you can create an interface with the right pkey automatically. The interface is also automatically switch to connected mode if IPOIB_CM is selected in the openibd.conf file. However, by default (at least from what I've seen on my system), the MTU of a newly created pkey specific ipoib interface is set to 2048 (due to the fact it is in datagram mode). Switching it to connected allows to get a much higher MTU but doesn't change it. Is there a way to set a default MTU for those interfaces? It should'nt be too hard to add something in the associated ifcfg script but I'll only to do it if it is not done somewhere else. Regards Nicolas From devel at morey-chaisemartin.com Tue Mar 31 04:08:39 2009 From: devel at morey-chaisemartin.com (Nicolas Morey-Chaisemartin) Date: Tue, 31 Mar 2009 13:08:39 +0200 Subject: [ofa-general] Setting MTU on ib0.????? In-Reply-To: <49D1F88C.5080501@ext.bull.net> References: <49D1F88C.5080501@ext.bull.net> Message-ID: <49D1F9B7.80201@morey-chaisemartin.com> My bad, I don't know what I did wrong earlier but setting connected mode automatically switch the MTU to 65520. So there's nothing to do ;) Nicolas Nicolas Morey Chaisemartin a écrit : > Hi, > > I saw in openibd script that through creating a /etc/sysconfig/network-script/ifcfg-ib?.???? you can create an interface with the right pkey automatically. > The interface is also automatically switch to connected mode if IPOIB_CM is selected in the openibd.conf file. > However, by default (at least from what I've seen on my system), the MTU of a newly created pkey specific ipoib interface is set to 2048 (due to the fact it is in datagram mode). > Switching it to connected allows to get a much higher MTU but doesn't change it. > Is there a way to set a default MTU for those interfaces? > It should'nt be too hard to add something in the associated ifcfg script but I'll only to do it if it is not done somewhere else. > > Regards > > Nicolas > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > From ogerlitz at Voltaire.com Tue Mar 31 04:24:36 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Tue, 31 Mar 2009 14:24:36 +0300 Subject: [ofa-general] [PATCH] IB/mlx4: Use pgprot_writecombine() for BlueFlame pages In-Reply-To: <200903301117.32355.jackm@dev.mellanox.co.il> References: <15ddcffd0903291006g4b7549cfj1879dd67518f8bff@mail.gmail.com> <200903301117.32355.jackm@dev.mellanox.co.il> Message-ID: <49D1FD74.9040205@Voltaire.com> Jack Morgenstein wrote: > You can see the write combining driver code in file: > kernel_patches/fixes/mlx4_0010_add_wc.patch Yes, with this patch, ib_rdma_lat goes down to about 1.2us where without any patch or with Roland's patch it gives about 3.5us Or. From ogerlitz at Voltaire.com Tue Mar 31 04:27:46 2009 From: ogerlitz at Voltaire.com (Or Gerlitz) Date: Tue, 31 Mar 2009 14:27:46 +0300 Subject: [ofa-general] [PATCH] IB/mlx4: Use pgprot_writecombine() for BlueFlame pages In-Reply-To: <49D1E6CF.8050101@mellanox.co.il> References: <49CF18DF.8080807@voltaire.com> <49D06FAE.90907@Voltaire.com> <49D1E6CF.8050101@mellanox.co.il> Message-ID: <49D1FE32.10703@Voltaire.com> Tziporet Koren wrote: > The fast path is enabled only when inline in used. OK, so since IPoIB doesn't use inline posting, its seems we have another issue here, as far as I can tell with connectx I saw ipoib latency of about 12us using RHEL 5.x kernels installed with ofed, where on the same nodes with 2.6.29 I get 19us - Roland, any suggestion how to resolve that? maybe as comparison you would like to see the numbers with the upstream 2.6.18 kernel? Or. From dorfman.eli at gmail.com Tue Mar 31 04:55:14 2009 From: dorfman.eli at gmail.com (Eli Dorfman (Voltaire)) Date: Tue, 31 Mar 2009 14:55:14 +0300 Subject: ***SPAM*** Re: ***SPAM*** Re: [ofa-general] OpenSM and trap 128. In-Reply-To: References: <49CB7723.9080104@ext.bull.net> <49CF6717.8010904@gmail.com> Message-ID: <49D204A2.6010802@gmail.com> Hal Rosenstock wrote: > On Sun, Mar 29, 2009 at 8:18 AM, Eli Dorfman (Voltaire) > wrote: > > > >> You can change sweep_on_trap option in opensm.conf to FALSE. >> This should stop opensm heavy sweeps. > > Yes but is that what is really desired ? If sweep on trap is disabled, > there is no reaction to legitimate subnet changes. > opensm will still react to subnet changes, but it will be delayed till next light sweep. this will ensure that opensm is not configuring all the time but only every sweep interval. Eli From hal.rosenstock at gmail.com Tue Mar 31 07:24:51 2009 From: hal.rosenstock at gmail.com (Hal Rosenstock) Date: Tue, 31 Mar 2009 10:24:51 -0400 Subject: ***SPAM*** Re: ***SPAM*** Re: [ofa-general] OpenSM and trap 128. In-Reply-To: <49D204A2.6010802@gmail.com> References: <49CB7723.9080104@ext.bull.net> <49CF6717.8010904@gmail.com> <49D204A2.6010802@gmail.com> Message-ID: On Tue, Mar 31, 2009 at 7:55 AM, Eli Dorfman (Voltaire) wrote: > Hal Rosenstock wrote: >> On Sun, Mar 29, 2009 at 8:18 AM, Eli Dorfman (Voltaire) >> wrote: >> >> >> >>> You can change sweep_on_trap option in opensm.conf to FALSE. >>> This should stop opensm heavy sweeps. >> >> Yes but is that what is really desired ? If sweep on trap is disabled, >> there is no reaction to legitimate subnet changes. >> > > opensm will still react to subnet changes, but it will be delayed till next light sweep. > this will ensure that opensm is not configuring all the time but only every sweep interval. switchinfo changes would be detected but other changes in the subnet only detected via heavy sweep would be missed, right ? If so, it seems reasonable to me to not have to disable the heavy sweep if a proper policy could be determined but the real solution is compliant SMA firmware. -- Hal > Eli > > From rdreier at cisco.com Tue Mar 31 08:18:00 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 31 Mar 2009 08:18:00 -0700 Subject: [ofa-general] [PATCH] IB/mlx4: Use pgprot_writecombine() for BlueFlame pages In-Reply-To: <49D1FD74.9040205@Voltaire.com> (Or Gerlitz's message of "Tue, 31 Mar 2009 14:24:36 +0300") References: <15ddcffd0903291006g4b7549cfj1879dd67518f8bff@mail.gmail.com> <200903301117.32355.jackm@dev.mellanox.co.il> <49D1FD74.9040205@Voltaire.com> Message-ID: > Yes, with this patch, ib_rdma_lat goes down to about 1.2us where without any patch or with Roland's patch it gives about 3.5us So the OFED WC patch is work 2.3usec?? Or did you mean 1.2usec vs. 1.5usec? In any case it seems my patch is not working on your system. Do you have CONFIG_X86_PAT set in your kernel .config? - R. From monis at Voltaire.COM Tue Mar 31 09:14:53 2009 From: monis at Voltaire.COM (Moni Shoua) Date: Tue, 31 Mar 2009 19:14:53 +0300 Subject: [ofa-general] [PATCH] rdma_cm: Add proc entry to monitor rdma_cm connections Message-ID: <49D2417D.9000608@Voltaire.COM> For each rdma_cm_id that is attached to a device print a set of fields that describe the connection that this id represents. Below is an example of the output of 'cat /proc/rdma_cm' This example is for a host with that runs a rping server and a rping client. TP DEV PO NDEV SRC DST PS ST QPN 0 mthca0 0 0.0.0.0:7174 262 8 0 1 mthca0 1 ib0 192.30.3.249:34478 192.30.3.248:7174 262 5 328710 1 mthca0 1 ib0 192.30.3.249:7174 192.30.3.248:47625 262 5 328711 Signed-off-by: Moni Shoua -- diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c index 2a2e508..5e5d4f2 100644 --- a/drivers/infiniband/core/cma.c +++ b/drivers/infiniband/core/cma.c @@ -50,6 +50,9 @@ #include #include #include +#ifdef CONFIG_PROC_FS +#include +#endif MODULE_AUTHOR("Sean Hefty"); MODULE_DESCRIPTION("Generic RDMA CM Agent"); @@ -2943,10 +2946,93 @@ static void cma_remove_one(struct ib_device *device) kfree(cma_dev); } + +#ifdef CONFIG_PROC_FS +static void proc_format_addr(struct sockaddr *sa, char* buf) +{ + switch (sa->sa_family) { + case AF_INET: { + struct sockaddr_in *sin = (struct sockaddr_in *)sa; + sprintf(buf, "%pI4:%u", &sin->sin_addr.s_addr, + be16_to_cpu(cma_port(sa))); + break; + } + case AF_INET6: { + struct sockaddr_in6 *sin6 = (struct sockaddr_in6 *)sa; + sprintf(buf, "%pI6:%u", &sin6->sin6_addr, + be16_to_cpu(cma_port(sa))); + break; + } + default: + buf[0] = 0; + } +} + +static int rdma_read_proc(char *page, char **start, off_t off, int count, + int *eof, void *data) +{ + int cc = 0; + char *cp = page; + struct rdma_id_private *id_priv; + char local_addr[64], remote_addr[64]; + + struct cma_device *cma_dev; + *cp = 0; + cc = sprintf(cp, + "%-3s" + "%-8s" + "%-3s" + "%-5s" + "%-52s" + "%-52s" + "%-5s" + "%-3s" + "%-8s" + "\n", + "TP", "DEV", "PO", "NDEV", "SRC", "DST", "PS", "ST", "QPN"); + cp += cc; + + mutex_lock(&lock); + list_for_each_entry(cma_dev, &dev_list, list) + list_for_each_entry(id_priv, &cma_dev->id_list, list) { + proc_format_addr((struct sockaddr *)&id_priv->id.route.addr.src_addr, + local_addr); + proc_format_addr((struct sockaddr *)&id_priv->id.route.addr.dst_addr, + remote_addr); + cc = sprintf(cp, + "%-3d" + "%-8s" + "%-3d" + "%-5s" + "%-52s" + "%-52s" + "%-5d" + "%-3d" + "%-8d" + "\n", + id_priv->id.route.addr.dev_addr.dev_type, + (id_priv->id.device) ? id_priv->id.device->name : "", + id_priv->id.port_num, + (id_priv->id.route.addr.dev_addr.src_dev) ? id_priv->id.route.addr.dev_addr.src_dev->name : "", + local_addr, + remote_addr, + id_priv->id.ps, + id_priv->state, + id_priv->qp_num); + cp += cc; + } + mutex_unlock(&lock); + *eof = 1; + return strlen(page); +} +#endif static int cma_init(void) { int ret, low, high, remaining; +#ifdef CONFIG_PROC_FS + struct proc_dir_entry *rdma_proc; +#endif get_random_bytes(&next_port, sizeof next_port); inet_get_local_port_range(&low, &high); remaining = (high - low) + 1; @@ -2963,6 +3049,15 @@ static int cma_init(void) ret = ib_register_client(&cma_client); if (ret) goto err; +#ifdef CONFIG_PROC_FS + rdma_proc = create_proc_entry("rdma_cm", 0444, NULL); + if (rdma_proc == NULL) { + printk(KERN_DEBUG "cannot create /proc/rdma_cm\n"); + } else { + rdma_proc->owner = THIS_MODULE; + rdma_proc->read_proc = rdma_read_proc; + } +#endif return 0; err: @@ -2984,6 +3079,9 @@ static void cma_cleanup(void) idr_destroy(&tcp_ps); idr_destroy(&udp_ps); idr_destroy(&ipoib_ps); +#ifdef CONFIG_PROC_FS + remove_proc_entry("rdma_cm", NULL); +#endif } module_init(cma_init); From rdreier at cisco.com Tue Mar 31 09:42:53 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 31 Mar 2009 09:42:53 -0700 Subject: [ofa-general] Re: [PATCH] rdma_cm: Add proc entry to monitor rdma_cm connections In-Reply-To: <49D2417D.9000608@Voltaire.COM> (Moni Shoua's message of "Tue, 31 Mar 2009 19:14:53 +0300") References: <49D2417D.9000608@Voltaire.COM> Message-ID: Umm, no... /proc is not the place for this, since it has nothing to do with processes. debugfs is probably the simplest place to put this info, unless you want to do something like netlink + a userspace tool. - R. From rdreier at cisco.com Tue Mar 31 09:47:11 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 31 Mar 2009 09:47:11 -0700 Subject: [ofa-general] Re: [PATCH] rdma_cm: Add proc entry to monitor rdma_cm connections In-Reply-To: <49D2417D.9000608@Voltaire.COM> (Moni Shoua's message of "Tue, 31 Mar 2009 19:14:53 +0300") References: <49D2417D.9000608@Voltaire.COM> Message-ID: By the way, if you have more than PAGE_SIZE of output from your read_proc function, then I think your code has a buffer overrun anyway... From rdreier at cisco.com Tue Mar 31 09:57:00 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 31 Mar 2009 09:57:00 -0700 Subject: [ofa-general] [PATCH] mlx4_core: Don't leak mailbox for SET_PORT on Ethernet ports Message-ID: Commit 793730bf ("mlx4_core: Don't perform SET_PORT command for Ethernet ports") introduced a leak of mailbox buffers when SET_PORT was called for Ethernet ports, since it added a return after the mailbox was allocated. Fix this by checking the port type and returning *before* allocating the mailbox. Signed-off-by: Roland Dreier --- I'll merge this for 2.6.30 unless something looks wrong to anyone... drivers/net/mlx4/port.c | 5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/net/mlx4/port.c b/drivers/net/mlx4/port.c index 7cce334..606aa58 100644 --- a/drivers/net/mlx4/port.c +++ b/drivers/net/mlx4/port.c @@ -299,13 +299,14 @@ int mlx4_SET_PORT(struct mlx4_dev *dev, u8 port) struct mlx4_cmd_mailbox *mailbox; int err; + if (dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH) + return 0; + mailbox = mlx4_alloc_cmd_mailbox(dev); if (IS_ERR(mailbox)) return PTR_ERR(mailbox); memset(mailbox->buf, 0, 256); - if (dev->caps.port_type[port] == MLX4_PORT_TYPE_ETH) - return 0; ((__be32 *) mailbox->buf)[1] = dev->caps.ib_port_def_cap[port]; err = mlx4_cmd(dev, mailbox->dma, port, 0, MLX4_CMD_SET_PORT, -- 1.6.0.4 From monis at Voltaire.COM Tue Mar 31 10:01:00 2009 From: monis at Voltaire.COM (Moni Shoua) Date: Tue, 31 Mar 2009 20:01:00 +0300 Subject: [ofa-general] Re: [PATCH] rdma_cm: Add proc entry to monitor rdma_cm connections In-Reply-To: References: <49D2417D.9000608@Voltaire.COM> Message-ID: <49D24C4C.2060203@Voltaire.COM> Roland Dreier wrote: > Umm, no... /proc is not the place for this, since it has nothing to do > with processes. > > debugfs is probably the simplest place to put this info, unless you want > to do something like netlink + a userspace tool. > > - R. > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > What avout /proc/net? I think it's very similar to /proc/net/tcp From rdreier at cisco.com Tue Mar 31 10:03:41 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 31 Mar 2009 10:03:41 -0700 Subject: [ofa-general] Re: [PATCH] rdma_cm: Add proc entry to monitor rdma_cm connections In-Reply-To: <49D24C4C.2060203@Voltaire.COM> (Moni Shoua's message of "Tue, 31 Mar 2009 20:01:00 +0300") References: <49D2417D.9000608@Voltaire.COM> <49D24C4C.2060203@Voltaire.COM> Message-ID: > What avout /proc/net? > I think it's very similar to /proc/net/tcp That's a legacy interface. The upstream kernel has a very strong policy that no more non-process-related stuff should go into /proc. - R. From rdreier at cisco.com Tue Mar 31 10:19:06 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 31 Mar 2009 10:19:06 -0700 Subject: [ofa-general] [PATCH] ipoib: fix free_netdev() BUG while deleting a vlan In-Reply-To: <49D10374.2090505@Voltaire.COM> (Yossi Etigin's message of "Mon, 30 Mar 2009 20:37:56 +0300") References: <49D10374.2090505@Voltaire.COM> Message-ID: > free_netdev() should be called after rtnl_lock is released. > Failing to to so might trigger an BUG in free_netdev() which asserts > that the device state is NETREG_UNREGISTERED. Thanks, good debugging. I like that patch below a little better since it has less duplication of unlocking on the exit path. However I think it was my fault that this got introduced in the first place. Can you let me know how the patch below looks to you? diff --git a/drivers/infiniband/ulp/ipoib/ipoib_vlan.c b/drivers/infiniband/ulp/ipoib/ipoib_vlan.c index 5a76a55..4c57f32 100644 --- a/drivers/infiniband/ulp/ipoib/ipoib_vlan.c +++ b/drivers/infiniband/ulp/ipoib/ipoib_vlan.c @@ -70,12 +70,14 @@ int ipoib_vlan_add(struct net_device *pdev, unsigned short pkey) */ if (ppriv->pkey == pkey) { result = -ENOTUNIQ; + priv = NULL; goto err; } list_for_each_entry(priv, &ppriv->child_intfs, list) { if (priv->pkey == pkey) { result = -ENOTUNIQ; + priv = NULL; goto err; } } @@ -96,7 +98,7 @@ int ipoib_vlan_add(struct net_device *pdev, unsigned short pkey) result = ipoib_set_dev_features(priv, ppriv->ca); if (result) - goto device_init_failed; + goto err; priv->pkey = pkey; @@ -109,7 +111,7 @@ int ipoib_vlan_add(struct net_device *pdev, unsigned short pkey) ipoib_warn(ppriv, "failed to initialize subinterface: " "device %s, port %d", ppriv->ca->name, ppriv->port); - goto device_init_failed; + goto err; } result = register_netdevice(priv->dev); @@ -146,19 +148,19 @@ sysfs_failed: register_failed: ipoib_dev_cleanup(priv->dev); -device_init_failed: - free_netdev(priv->dev); - err: mutex_unlock(&ppriv->vlan_mutex); rtnl_unlock(); + if (priv) + free_netdev(priv->dev); + return result; } int ipoib_vlan_delete(struct net_device *pdev, unsigned short pkey) { struct ipoib_dev_priv *ppriv, *priv, *tpriv; - int ret = -ENOENT; + struct net_device *dev = NULL; if (!capable(CAP_NET_ADMIN)) return -EPERM; @@ -172,14 +174,17 @@ int ipoib_vlan_delete(struct net_device *pdev, unsigned short pkey) unregister_netdevice(priv->dev); ipoib_dev_cleanup(priv->dev); list_del(&priv->list); - free_netdev(priv->dev); - - ret = 0; + dev = priv->dev; break; } } mutex_unlock(&ppriv->vlan_mutex); rtnl_unlock(); - return ret; + if (dev) { + free_netdev(dev); + return 0; + } + + return -ENODEV; } From jgunthorpe at obsidianresearch.com Tue Mar 31 10:46:18 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Tue, 31 Mar 2009 11:46:18 -0600 Subject: [ofa-general] Re: [PATCH] rdma_cm: Add proc entry to monitor rdma_cm connections In-Reply-To: References: <49D2417D.9000608@Voltaire.COM> <49D24C4C.2060203@Voltaire.COM> Message-ID: <20090331174618.GB32482@obsidianresearch.com> On Tue, Mar 31, 2009 at 10:03:41AM -0700, Roland Dreier wrote: > > What avout /proc/net? > > I think it's very similar to /proc/net/tcp > > That's a legacy interface. The upstream kernel has a very strong policy > that no more non-process-related stuff should go into /proc. AFAIK, proc/net is the only source for most of the information under proc/net, it can't be legacy if there is no alternative :) These days it is actually proc//net/XXX due the namespace stuff, so it kinda is process related. People are adding new things, ie proc/net/sctp/remaddr was added last year. /proc/net/rdma/ really does seem to be the right answer today :| debugfs is no good because tools like lsof/netstat need to access this information and distros don't mount it by default. BTW, Moni, while you are looking at this it would be really nice to have a proc//qp directory or file so that lists of QPs associated with processes can be produced by lsof. Regards, Jason From rdreier at cisco.com Tue Mar 31 10:57:06 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 31 Mar 2009 10:57:06 -0700 Subject: [ofa-general] Re: [PATCH] rdma_cm: Add proc entry to monitor rdma_cm connections In-Reply-To: <20090331174618.GB32482@obsidianresearch.com> (Jason Gunthorpe's message of "Tue, 31 Mar 2009 11:46:18 -0600") References: <49D2417D.9000608@Voltaire.COM> <49D24C4C.2060203@Voltaire.COM> <20090331174618.GB32482@obsidianresearch.com> Message-ID: > AFAIK, proc/net is the only source for most of the information under > proc/net, it can't be legacy if there is no alternative :) These days > it is actually proc//net/XXX due the namespace stuff, so it kinda > is process related. Well at least /proc/net/tcp has been deprecated for a *long* time (5+ years?) in favor of tcp_diag -- netstat vs. ss is basically the same issue as ifconfig vs. ip. If you want a model, tcp_diag is much better than the crufty /proc/net/tcp code. - R. From rdreier at cisco.com Tue Mar 31 10:58:13 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 31 Mar 2009 10:58:13 -0700 Subject: [ofa-general] Re: [PATCH] rdma_cm: Add proc entry to monitor rdma_cm connections In-Reply-To: (Roland Dreier's message of "Tue, 31 Mar 2009 10:57:06 -0700") References: <49D2417D.9000608@Voltaire.COM> <49D24C4C.2060203@Voltaire.COM> <20090331174618.GB32482@obsidianresearch.com> Message-ID: > Well at least /proc/net/tcp has been deprecated for a *long* time (5+ > years?) in favor of tcp_diag -- netstat vs. ss is basically the same > issue as ifconfig vs. ip. And btw "ss -e" shows info not available in /proc/net/tcp. From arlin.r.davis at intel.com Tue Mar 31 11:41:22 2009 From: arlin.r.davis at intel.com (Arlin Davis) Date: Tue, 31 Mar 2009 11:41:22 -0700 Subject: [ofa-general] [PATCH 1/1] dapl (v2.0): ia64 build problem on SuSE 11, atomic.h no longer exists. Message-ID: <000301c9b230$4d2deec0$66d5180a@amr.corp.intel.com> Add autotools check for SuSE 11 and include intrinsics.h Signed-off-by: Arlin Davis --- Makefile.am | 4 ++++ configure.in | 10 ++++++++++ dapl/udapl/linux/dapl_osd.h | 7 ++++++- 3 files changed, 20 insertions(+), 1 deletions(-) diff --git a/Makefile.am b/Makefile.am index 08a8cce..721996f 100755 --- a/Makefile.am +++ b/Makefile.am @@ -11,6 +11,10 @@ if OS_RHEL5 OSFLAGS += -DREDHAT_EL5 endif +if OS_SUSE11 +OSFLAGS += -DSUSE_11 +endif + if EXT_TYPE_IB XFLAGS = -DDAT_EXTENSIONS XPROGRAMS_CMA = dapl/openib_cma/dapl_ib_extensions.c diff --git a/configure.in b/configure.in index 8401d29..8220893 100644 --- a/configure.in +++ b/configure.in @@ -86,6 +86,16 @@ AC_CACHE_CHECK(Check for RHEL5 system, ac_cv_rhel5, fi) AM_CONDITIONAL(OS_RHEL5, test "$ac_cv_rhel5" = "yes") +dnl Check for SuSE release 11 +AC_CACHE_CHECK(Check for SUSE_11 system, ac_cv_suse11, + if test -f /etc/SuSE-release && + test -n "`grep -e "VERSION = 11" /etc/SuSE-release`"; then + ac_cv_suse11=yes + else + ac_cv_suse11=no + fi) +AM_CONDITIONAL(OS_SUSE11, test "$ac_cv_suse11" = "yes") + AC_CONFIG_FILES([Makefile test/dtest/Makefile test/dapltest/Makefile dapl.spec]) AC_OUTPUT diff --git a/dapl/udapl/linux/dapl_osd.h b/dapl/udapl/linux/dapl_osd.h index 974a0f0..1c098c5 100644 --- a/dapl/udapl/linux/dapl_osd.h +++ b/dapl/udapl/linux/dapl_osd.h @@ -92,10 +92,15 @@ #include #include -#if !defined(REDHAT_EL5) && (defined(__ia64__)) +#if !defined(SUSE_11) && !defined(REDHAT_EL5) && defined(__ia64__) #include #endif +#if defined(SUSE_11) && defined(__ia64__) +#include +#include +#endif + /* Useful debug definitions */ #ifndef STATIC #define STATIC static -- 1.5.2.5 From jgunthorpe at obsidianresearch.com Tue Mar 31 11:58:49 2009 From: jgunthorpe at obsidianresearch.com (Jason Gunthorpe) Date: Tue, 31 Mar 2009 12:58:49 -0600 Subject: [ofa-general] Re: [PATCH] rdma_cm: Add proc entry to monitor rdma_cm connections In-Reply-To: References: <49D2417D.9000608@Voltaire.COM> <49D24C4C.2060203@Voltaire.COM> <20090331174618.GB32482@obsidianresearch.com> Message-ID: <20090331185849.GD32482@obsidianresearch.com> On Tue, Mar 31, 2009 at 10:57:06AM -0700, Roland Dreier wrote: > > AFAIK, proc/net is the only source for most of the information under > > proc/net, it can't be legacy if there is no alternative :) These days > > it is actually proc//net/XXX due the namespace stuff, so it kinda > > is process related. > > Well at least /proc/net/tcp has been deprecated for a *long* time (5+ > years?) in favor of tcp_diag -- netstat vs. ss is basically the same > issue as ifconfig vs. ip. Well, squeezing things through the INET_DIAG netlink interface would be nice, but several other modern things have not bothered - like sctp, or even netlink itself, they use proc/net exclusively. (not that I think that is ideal, but..) Also keep in mind, patching things like lsof and netstat to use proc/net files is alot simpler than trying to work netlink into those programs for the first time.. Jason From swise at opengridcomputing.com Tue Mar 31 12:06:37 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Tue, 31 Mar 2009 14:06:37 -0500 Subject: [ofa-general] Re: [PATCH] rdma_cm: Add proc entry to monitor rdma_cm connections In-Reply-To: References: <49D2417D.9000608@Voltaire.COM> <49D24C4C.2060203@Voltaire.COM> <20090331174618.GB32482@obsidianresearch.com> Message-ID: <49D269BD.4050909@opengridcomputing.com> Roland Dreier wrote: > > Well at least /proc/net/tcp has been deprecated for a *long* time (5+ > > years?) in favor of tcp_diag -- netstat vs. ss is basically the same > > issue as ifconfig vs. ip. > > And btw "ss -e" shows info not available in /proc/net/tcp. > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > I think we definitely need something like this, and an rdmastat command to do useful formatting... What about putting this stuff in /sys/class/infiniband/rdma_cm/* You could populate a file or directory for each cm_id and reading the file (or files under the directory) could display the various useful state. Steve From swise at opengridcomputing.com Tue Mar 31 12:12:10 2009 From: swise at opengridcomputing.com (Steve Wise) Date: Tue, 31 Mar 2009 14:12:10 -0500 Subject: [ofa-general] Re: [PATCH] rdma_cm: Add proc entry to monitor rdma_cm connections In-Reply-To: <49D269BD.4050909@opengridcomputing.com> References: <49D2417D.9000608@Voltaire.COM> <49D24C4C.2060203@Voltaire.COM> <20090331174618.GB32482@obsidianresearch.com> <49D269BD.4050909@opengridcomputing.com> Message-ID: <49D26B0A.8020802@opengridcomputing.com> > What about putting this stuff in /sys/class/infiniband/rdma_cm/* > > > You could populate a file or directory for each cm_id and reading the > file (or files under the directory) could display the various useful > state. > > Here's an example where each cm_id is a directory tree under /sys/class/infiniband/rdma_cm/* Two allocated cm_ids might take dir entries "0" and "1" (for lack of a better token): /sys/class/infiniband/rdma_cm/0/ /sys/class/infiniband/rdma_cm/1/ Then under these are files: qp_num src_addr src_port dst_addr dst_port port_space device_name id_state ib/* (for ib-specific stuff) iw/* (for iw-specific stuff) Steve. From arlin.r.davis at intel.com Tue Mar 31 12:16:23 2009 From: arlin.r.davis at intel.com (Davis, Arlin R) Date: Tue, 31 Mar 2009 12:16:23 -0700 Subject: [ofa-general] [ANNOUNCE] dapl-1.2.14 and dapl-2.0.17 release Message-ID: New release for dapl 1.2 and 2.0 available on the OFA download page and in my git tree. md5sum: f58d6dd903cee271d71b0eb6fa33984e compat-dapl-1.2.14.tar.gz md5sum: 617ebf54456b5559ea49e787515a8fa8 dapl-2.0.17.tar.gz Summary of changes: v1,v2 - Fix SuSE 11 build issue, asm/atomic.h no longer exists (Bug #1564) Vlad, please pull both packages into OFED 1.4.1 RC3 and install the following: compat-dapl-1.2.14-1 compat-dapl-devel-1.2.14-1 dapl-2.0.17-1 dapl-utils-2.0.17-1 dapl-devel-2.0.17-1 dapl-debuginfo-2.0.17-1 See http://www.openfabrics.org/downloads/dapl/ more details. -arlin From rdreier at cisco.com Tue Mar 31 13:25:57 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 31 Mar 2009 13:25:57 -0700 Subject: [ofa-general] Re: [PATCH] rdma_cm: Add proc entry to monitor rdma_cm connections In-Reply-To: <49D269BD.4050909@opengridcomputing.com> (Steve Wise's message of "Tue, 31 Mar 2009 14:06:37 -0500") References: <49D2417D.9000608@Voltaire.COM> <49D24C4C.2060203@Voltaire.COM> <20090331174618.GB32482@obsidianresearch.com> <49D269BD.4050909@opengridcomputing.com> Message-ID: > You could populate a file or directory for each cm_id and reading the > file (or files under the directory) could display the various useful > state. Having a sysfs file per connection is huge overhead, both in terms of the number of syscalls to read it, and also the pinned kernel memory just for the dentries etc. I don't think it's a great idea. - R. From rdreier at cisco.com Tue Mar 31 13:29:21 2009 From: rdreier at cisco.com (Roland Dreier) Date: Tue, 31 Mar 2009 13:29:21 -0700 Subject: [ofa-general] Re: [PATCH] rdma_cm: Add proc entry to monitor rdma_cm connections In-Reply-To: <20090331185849.GD32482@obsidianresearch.com> (Jason Gunthorpe's message of "Tue, 31 Mar 2009 12:58:49 -0600") References: <49D2417D.9000608@Voltaire.COM> <49D24C4C.2060203@Voltaire.COM> <20090331174618.GB32482@obsidianresearch.com> <20090331185849.GD32482@obsidianresearch.com> Message-ID: > Well, squeezing things through the INET_DIAG netlink interface would > be nice, but several other modern things have not bothered - like sctp, > or even netlink itself, they use proc/net exclusively. (not that I > think that is ideal, but..) Tell you what -- if you can get consensus from netdev about adding rdma connection stats to /proc/net then I withdraw my objection. But I really thing everyone is going to agree that adding more proc files is not a good idea. > Also keep in mind, patching things like lsof and netstat to use > proc/net files is alot simpler than trying to work netlink into those > programs for the first time.. OK but eg ifconfig already doesn't work for IPoIB -- and the answer is "just use ip instead if you care". - R. From jeff.scott at openfabrics.org Tue Mar 31 14:19:34 2009 From: jeff.scott at openfabrics.org (Jeffrey Scott) Date: Tue, 31 Mar 2009 14:19:34 -0700 Subject: [ofa-general] Sonoma presentations are on the web Message-ID: You can view the presentations from Sonoma at http://www.openfabrics.org/archives/sonoma2009.htm. A few presentations aren't up yet. We hope to add them soon. ---------------------------------------- Jeffrey Scott OpenFabrics Alliance Office 408-884-4017 Mobile 408-348-3651 www.OpenFabrics.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsquyres at cisco.com Tue Mar 31 14:27:04 2009 From: jsquyres at cisco.com (Jeff Squyres) Date: Tue, 31 Mar 2009 17:27:04 -0400 Subject: [ofa-general] Sonoma presentations are on the web In-Reply-To: References: Message-ID: Jeff -- I don't see the panel slides listed. Do you plan to post those as well? On Mar 31, 2009, at 5:19 PM, Jeffrey Scott wrote: > You can view the presentations from Sonoma at http://www.openfabrics.org/archives/sonoma2009.htm > . > > A few presentations aren’t up yet. We hope to add them soon. > > > ---------------------------------------- > Jeffrey Scott > OpenFabrics Alliance > > Office 408-884-4017 > Mobile 408-348-3651 > www.OpenFabrics.org > > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -- Jeff Squyres Cisco Systems From twbowman at gmail.com Tue Mar 31 15:13:22 2009 From: twbowman at gmail.com (Todd Bowman) Date: Tue, 31 Mar 2009 16:13:22 -0600 Subject: ***SPAM*** Re: [ofa-general] Sonoma presentations are on the web In-Reply-To: References: Message-ID: I understand that this is not completed, but one of the links is broken: Traffic Management & Optimization for IB Thanks, Todd 2009/3/31 Jeff Squyres > Jeff -- I don't see the panel slides listed. Do you plan to post those as > well? > > > > On Mar 31, 2009, at 5:19 PM, Jeffrey Scott wrote: > > You can view the presentations from Sonoma at >> http://www.openfabrics.org/archives/sonoma2009.htm. >> >> A few presentations aren’t up yet. We hope to add them soon. >> >> >> ---------------------------------------- >> Jeffrey Scott >> OpenFabrics Alliance >> >> Office 408-884-4017 >> Mobile 408-348-3651 >> www.OpenFabrics.org >> >> >> _______________________________________________ >> general mailing list >> general at lists.openfabrics.org >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general >> >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> > > > -- > Jeff Squyres > Cisco Systems > > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeff at splitrockpr.com Tue Mar 31 15:19:45 2009 From: jeff at splitrockpr.com (Jeffrey Scott) Date: Tue, 31 Mar 2009 15:19:45 -0700 Subject: [ofa-general] Sonoma presentations are on the web In-Reply-To: References: Message-ID: <9B2B809297944A49B9FB86579DEB451A@Gaucho> Thanks. I'll work on it. Note to everyone, if you have questions about the slides, just send an email to me directly. No need to cc the email lists. _____ From: Todd Bowman [mailto:twbowman at gmail.com] Sent: Tuesday, March 31, 2009 3:13 PM To: Jeff Squyres Cc: Jeffrey Scott; OFA Marketing Working Group; general at lists.openfabrics.org Subject: Re: [ofa-general] Sonoma presentations are on the web I understand that this is not completed, but one of the links is broken: Traffic Management & Optimization for IB Thanks, Todd 2009/3/31 Jeff Squyres Jeff -- I don't see the panel slides listed. Do you plan to post those as well? On Mar 31, 2009, at 5:19 PM, Jeffrey Scott wrote: You can view the presentations from Sonoma at http://www.openfabrics.org/archives/sonoma2009.htm. A few presentations aren't up yet. We hope to add them soon. ---------------------------------------- Jeffrey Scott OpenFabrics Alliance Office 408-884-4017 Mobile 408-348-3651 www.OpenFabrics.org _______________________________________________ general mailing list general at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -- Jeff Squyres Cisco Systems _______________________________________________ general mailing list general at lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jie.Cai at cs.anu.edu.au Tue Mar 31 18:01:52 2009 From: Jie.Cai at cs.anu.edu.au (Jie Cai) Date: Wed, 01 Apr 2009 12:01:52 +1100 Subject: [ofa-general] uDAPL DTO completion question. Message-ID: <49D2BD00.5010002@cs.anu.edu.au> Local side dat_ep_post_send message to remote side, with DAT_COMPLETION_DEFAULT_FLAG. Then the local side dat_evd_wait for the send completes. Does the DTO completion event at local side of the send indicates the data has been received at remote side as well? I am bit confused about this. -- Jie Cai From caitlin.bestler at gmail.com Tue Mar 31 20:40:12 2009 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Tue, 31 Mar 2009 20:40:12 -0700 Subject: ***SPAM*** Re: [ofa-general] uDAPL DTO completion question. In-Reply-To: <49D2BD00.5010002@cs.anu.edu.au> References: <49D2BD00.5010002@cs.anu.edu.au> Message-ID: <469958e00903312040j7700d2ccr9104996c2fc29cd4@mail.gmail.com> On Tue, Mar 31, 2009 at 6:01 PM, Jie Cai wrote: > Local side dat_ep_post_send message to remote side, > with DAT_COMPLETION_DEFAULT_FLAG. > Then the local side dat_evd_wait for the send completes. > > Does the DTO completion event at local side of the > send indicates the data has been received at remote side > as well? > > I am bit confused about this. > A DAT DTO send completion means that the DAT Provider and its underlying Transport no longer require your buffers -- and that barring an error that tears down the connection, the data has been or will be delivered to the remote peer. This may mean as little as the transport layer has linearized your send into its own buffer. Since most transport providers are not eager to volunteer vast amounts of transmit buffering it will frequently mean s that the remote host has acked the data. That is the lowest cost method for the sender to know that the source buffers are no longer needed. Even when the underlying transport guarantees that the data is in the remote host's buffers, it does NOT guarantee that it is in the remote peer's cache or that the remote peer has been scheduled or has in any way noticed that the memory has been updated. The only guarantee that the remote peer *application* has gotten your message is when it responds to it. From arkady.kanevsky at gmail.com Tue Mar 31 22:53:36 2009 From: arkady.kanevsky at gmail.com (arkady kanevsky) Date: Wed, 1 Apr 2009 01:53:36 -0400 Subject: ***SPAM*** Re: [ofa-general] uDAPL DTO completion question. In-Reply-To: <469958e00903312040j7700d2ccr9104996c2fc29cd4@mail.gmail.com> References: <49D2BD00.5010002@cs.anu.edu.au> <469958e00903312040j7700d2ccr9104996c2fc29cd4@mail.gmail.com> Message-ID: <517c62fb0903312253w6344d62j1b8c072354b15ad2@mail.gmail.com> Correct. The only guarantee you have when you get Send completion is that you can reuse the buffer and message has been sent. It does not guarantee that message is in recv buffer. For some transports, like IB, which have ack/nack it does provide extra guarantee that it it reach remote NIC. But even in this case it does not guarantee that it is in user remote buffer. For these type of guarantee you need application level ack. But there is one more guarantee that Send competion provides. If data will not make it to remote side buffer the connection will be broken before match recv on the other side will complete. Moreover, recv on another side will not complete with success if the data had not arrived fully and without errors. Thanks, Arkady On Tue, Mar 31, 2009 at 11:40 PM, Caitlin Bestler wrote: > On Tue, Mar 31, 2009 at 6:01 PM, Jie Cai wrote: > > Local side dat_ep_post_send message to remote side, > > with DAT_COMPLETION_DEFAULT_FLAG. > > Then the local side dat_evd_wait for the send completes. > > > > Does the DTO completion event at local side of the > > send indicates the data has been received at remote side > > as well? > > > > I am bit confused about this. > > > A DAT DTO send completion means that the DAT Provider and its > underlying Transport no longer require your buffers -- and that barring > an error that tears down the connection, the data has been or will be > delivered to the remote peer. > > This may mean as little as the transport layer has linearized your send > into its own buffer. Since most transport providers are not eager to > volunteer vast amounts of transmit buffering it will frequently mean > s that the remote host has acked the data. That is the lowest cost > method for the sender to know that the source buffers are no longer > needed. > > Even when the underlying transport guarantees that the data is in the > remote host's buffers, it does NOT guarantee that it is in the remote > peer's cache or that the remote peer has been scheduled or has > in any way noticed that the memory has been updated. > > The only guarantee that the remote peer *application* has gotten your > message is when it responds to it. > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > -- Cheers, Arkady Kanevsky -------------- next part -------------- An HTML attachment was scrubbed... URL: From Jie.Cai at cs.anu.edu.au Tue Mar 31 23:41:03 2009 From: Jie.Cai at cs.anu.edu.au (Jie Cai) Date: Wed, 01 Apr 2009 17:41:03 +1100 Subject: [ofa-general] uDAPL DTO completion question. In-Reply-To: <517c62fb0903312253w6344d62j1b8c072354b15ad2@mail.gmail.com> References: <49D2BD00.5010002@cs.anu.edu.au> <469958e00903312040j7700d2ccr9104996c2fc29cd4@mail.gmail.com> <517c62fb0903312253w6344d62j1b8c072354b15ad2@mail.gmail.com> Message-ID: <49D30C7F.1050201@cs.anu.edu.au> Understood now. A further question is here again. To implement software level acknowledgment to inform initiator that data has been available for remoter, is that possible to use a busy loop at remote side to detect the last element of transferring has appear in the memory. Or remoter has to wait for the event of recv matching initiator's send, then send a message back to initiator as a acknowledgment? I have tried with busy loop implementation, but failed. Memory of last element does not changed at all. I guess DAPL may not allow user to modify the recv buffer when there is a incoming message into it. Please kindly clarify this. arkady kanevsky wrote: > Correct. > The only guarantee you have when you get Send completion is that > you can reuse the buffer and message has been sent. > It does not guarantee that message is in recv buffer. > > For some transports, like IB, which have ack/nack it does > provide extra guarantee that it it reach remote NIC. But > even in this case it does not guarantee that it is in user remote buffer. > > For these type of guarantee you need application level ack. > > But there is one more guarantee that Send competion provides. > If data will not make it to remote side buffer the connection will be > broken > before match recv on the other side will complete. Moreover, recv on > another side > will not complete with success if the data had not arrived fully and > without errors. > Thanks, > Arkady > > On Tue, Mar 31, 2009 at 11:40 PM, Caitlin Bestler > > wrote: > > On Tue, Mar 31, 2009 at 6:01 PM, Jie Cai > wrote: > > Local side dat_ep_post_send message to remote side, > > with DAT_COMPLETION_DEFAULT_FLAG. > > Then the local side dat_evd_wait for the send completes. > > > > Does the DTO completion event at local side of the > > send indicates the data has been received at remote side > > as well? > > > > I am bit confused about this. > > > A DAT DTO send completion means that the DAT Provider and its > underlying Transport no longer require your buffers -- and that > barring > an error that tears down the connection, the data has been or will be > delivered to the remote peer. > > This may mean as little as the transport layer has linearized your > send > into its own buffer. Since most transport providers are not eager to > volunteer vast amounts of transmit buffering it will frequently mean > s that the remote host has acked the data. That is the lowest cost > method for the sender to know that the source buffers are no longer > needed. > > Even when the underlying transport guarantees that the data is in the > remote host's buffers, it does NOT guarantee that it is in the remote > peer's cache or that the remote peer has been scheduled or has > in any way noticed that the memory has been updated. > > The only guarantee that the remote peer *application* has gotten your > message is when it responds to it. > _______________________________________________ > general mailing list > general at lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > > > > -- > Cheers, > Arkady Kanevsky